search

benbusby / whoogle-search

A self-hosted, ad-free, privacy-respecting metasearch engine
https://pypi.org/project/whoogle-search/
MIT License
9.11k stars 906 forks source link

[BUG] Using Whoogle to scrape Google search results is very difficult due to strange class names #183

Closed mendel5 closed 3 years ago

mendel5 commented 3 years ago

Describe the bug I am using Whoogle to scrape Google search results. The reason I am using Whoogle and not Google directly is because Google regularly asks my scraper to solve a Recaptcha.

My technical setup is Python with Selenium and Firefox as webdriver. The rendered HTML gets parsed with beautiful soup.

The naming scheme of the HTML classes on Whoogle's search result page makes it very difficult to scrape and parse what I am actually looking for. I would like to scrape the website's title, the description and the URL.

For example the URLs on the search results page are contained in <div class="kCrYT">. However there are other elements of the same class that don't contain a link. It would be very helpful for the <a href="" tag to have it's own class.

The website description is contained in <div class="BNeawe s3v9rd AP7Wnd">. The links to the previous page and next page are contained in the same class="nBDE1b G5eFlf". It would be helpful for both page links to have different classes or specific ids.

To Reproduce Steps to reproduce the behavior:

  1. Open Whoogle and search for something
  2. Open the browser's developer tools and inspect the HTML elements
  3. Be confused by strange HTML class names

Deployment Method Download git repo and use run executable

Version of Whoogle Search Whoogle Search v0.3.0

mendel5 commented 3 years ago

As far as I can see the strange HTML class names are originating from Google and Whoogle just inherits them?

benbusby commented 3 years ago

Yes, all classes are inherited from Google directly. They also change fairly frequently, as I've discovered in the past. It'd probably be better to use BeautifulSoup to look for patterns in the HTML rather than classes (i.e. extract all a tags with a h3 child, since that's how the results are formatted), but off the top of my head I'm not sure of an exact solution.

mendel5 commented 3 years ago

Yes, all classes are inherited from Google directly. They also change fairly frequently

That seems to be quite evil by Google...

It'd probably be better to use BeautifulSoup to look for patterns in the HTML rather than classes (i.e. extract all a tags with a h3 child, since that's how the results are formatted)

Thanks for the suggestion. I'll see what I can do.