Not all text matched with link nested in spans

[eht16]

With some HTML like:

<span class="h-card">
 <a class="u-url mention" href="http://localhost/" rel="nofollow noopener noreferrer" target="_blank">
  @
  <span>
   matched text
  </span>
 </a>
</span>
Some text which is missing.

only the text of the a tag is matched but not the text beyong (the HTML snippet corresponds to the returned content field in the Mastodon API, https://docs.joinmastodon.org/methods/statuses/#get).

I'm not sure if this is desired behavior or a bug. For comparison I tried BeautifulSoup and html2text and both consider the additional text and return it.

>>> import bs4, html2text, requests_html
>>> html = "<span class=\"h-card\"><a class=\"u-url mention\" href=\"http://localhost/\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">@<span>matched text</span></a></span> Some text which is missing."
>>> print(bs4.BeautifulSoup(html, features="html.parser").get_text('\n'))
@
matched text
 Some text which is missing.
>>> print(html2text.html2text(html))
[@matched text](http://localhost/) Some text which is missing.

>>> print(requests_html.HTML(html=html).text)
@matched text
>>> 

P.S.: great to see this package is maintained again! :heart:

[surister]

It wouldn't be the first time that I see this, for example the Scala HTML/XML parser implementation used by Spark in Azure has this behavior. Talked to Microsoft about it and it is intended behavior.

Parsing HTML is hard since pretty much almost any string is 'valid' HTML, anyway, without going too deep into the pyquery/lxml implementation I wouldn't be surprised if it was intended.