Improve scraper performance

[Narlotl]

Scraping all the domains can take a long time. Many of the domains aren't even active, but time is still spent on them.

• [ ] Check if domain is active without fetching content
• [ ] Combine URL and metadata scraping to reduce number of requests
• [ ] Don't check for sitemap/robots if the homepage doesn't respond