search

ecoron / SerpScrap

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
https://github.com/ecoron/SerpScrap
MIT License
257 stars 61 forks source link

100 results from google search #30

Closed tobyhage closed 6 years ago

tobyhage commented 6 years ago

Hello,

First thank you for this nice software :-).

I think i have found two issues. The first one is that i don't get 100 results from google search.

I have set following config options:

config.set('num_pages_for_keyword', 1) config.set('num_results_per_page', 100)

But i get 10 results back instead of 100.

Is this a known issue or do i miss something....

I will post the other issue i have found in another post.

kind regards,

Toby.

bernardosrulzon commented 6 years ago

@tobyhage Try this workaround:

config.set('google_search_url', 'https://www.google.com/search?num=100')
tobyhage commented 6 years ago

Thank you, this is working :-)

ecoron commented 6 years ago

@bernardosrulzon thx for sharing. i try to find some time to work on this project again.

ecoron commented 6 years ago

starting in 0.10.3 you can also just use the config key 'num_results_per_page' config.set('num_results_per_page', 100)