[Favicons Refactor] Some search engines return 404 when attempting to process HTML to scrape favicon URL

[data-sync-user]

Some search engines return a 404 when trying to scrape a favicon URL from the webpage’s HTML. This results in displaying a letter favicon fallback instead of the correct favicon.

This happens with crates.io for example.

In the case of crates.io, the solution seems to be to add the Accept header to requests: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Otherwise you get a 404 response for the search terms URL: https://crates.io/search?q=%7BsearchTerms%7D

QA Test Notes / How to reproduce:

1. Add a custom search engine for the first time (make sure this is an install where you have not visited crates.io before or used it as a search engine before)
2. Use this query string for crates.io: https://crates.io/search?q=%s
3. Observe the favicon that appears after the search engine has been added.

Expected result:

crates.io has a yellow blocky favicon. It should ﹍not﹍ show a letter favicon.

Good result:

!Screenshot 2024-10-10 at 1.56.44 PM.png|width=83.33333333333334%,alt="Screenshot 2024-10-10 at 1.56.44 PM.png"!

Github link

┆Issue is synchronized with this Jira Task

[data-sync-user]

➤ ih-codes commented:

RelMan:

This fix should be merged into the next release (which I believe will be v132).