search

tasos-py / Search-Engines-Scraper

Search google, bing, yahoo, and other search engines with python
MIT License
513 stars 137 forks source link

Suggestion: Collect ad based and organic links #67

Closed jatlantic closed 1 week ago

jatlantic commented 6 months ago

Hi,

Love this package and was wondering if it would be possible to add a feature where one could collect the ad based and organic urls/link results (for Bing for example)?

Cheers, jatlantic

tasos-py commented 6 months ago

Hi Jatlantic. While ads should be present in HTTP response bodies, the css selectors ere designed to ignore them. So, we should be able to change the selectors and get them, but I don't see the need to add this feature. Of course, you could just create a fork and add it yourself. And if you choose to do so I'll be glad to help

jatlantic commented 6 months ago

Hi tasos-py, thanks a lot for your swift answer. And yes it makes absolute sense. I'll do a fork to capture organic and ad results for Bing and will go from there.

tasos-py commented 6 months ago

That's awesome! To help you get started, you'll find SearchEngine objects in "/search_engines/engines/". Each of them has all required selectors in a ._selectors() method that you'd have to edit. Other than that you could maybe add a flag or argument in the parent object to chose between ads and organic results

jatlantic commented 6 months ago

Good advice, adapted things for a Bing search here: https://github.com/jatlantic/Search-Engines-Scraper. Likely you would do it more elegantly:) Happy to build in your suggestions.