Not all meta tags are found

[rouven-uncap]

Description

Parsing a Tweet html, Floki only returns the initial 11 <meta> tags. Notice in the document from twitter.com below, several <script> and <link> tags follow, and then additional <meta> tags are present. These are not found by Floki.

To Reproduce

Steps to reproduce the behavior:

• Using Floki v0.30.1
• Using Elixir v1.11.2
• Using Erlang OTP 23
• With this code:

  {:ok, %{body: body}} = HTTPoison.get("https://twitter.com/SpaceX/status/1389672850089418752") 
  {:ok, document} = Floki.parse_document(body)
  Floki.find(document, "meta")

  Expected behavior

All <meta> tags should be returned by the query

[philss]

@rouven-uncap for what I could see, they are all returned:

Maybe Twitter is returning a different result when downloading using HTTPoison. I downloaded the file using curl and then I opened in my iex. Please try again.

[rouven-uncap]

If you look further down below the link and script tags, there are more meta tags below, which are not returned in your screenshot:

[rouven-uncap]

You are correct about curl. Maybe those extra meta tags are injected into the DOM via JS after the page is loaded, which completely defeats the purpose of meta tags...