search

opsdisk / yagooglesearch

Yet another googlesearch - A Python library for executing intelligent, realistic-looking, and tunable Google searches.
BSD 3-Clause "New" or "Revised" License
241 stars 42 forks source link

Sometimes the search results are empty, how should I troubleshoot the issue? #36

Closed photkey closed 7 months ago

photkey commented 9 months ago

I used a proxy IP for the search, and sometimes when using the exact same keywords, the search results are empty, while other times there are results. I suspect it is due to the quality of the proxy IP, but I am not sure how to confirm the cause of the problem or how to resolve it. Can this library add a return similar to HTTP_429_DETECTED, which is a prompt returned in case of network issues, for example: HTTP_ERROR? This can be used in the script to handle this result, such as changing the proxy

photkey commented 9 months ago

My current solution is to use a new proxy for data collection if the result is empty. I will try this up to 10 times. This method can solve the current problem in most cases, but it is not perfect. For example, if there are keywords that do not have any results or if I encounter 10 problematic proxy IPs in a row. It would be best if yagooglesearch could visually indicate whether there is a network issue when returning results, similar to the mentioned HTTP_ERROR.

opsdisk commented 8 months ago

Hi @photkey apologies it took a while to get back to you. Two quick things...

Do you get back a specific HTTP error code (401, 404, etc.) when results are empty? The HTTP 429 is easy because that relates to interacting with Google itself. If empty results are returned, that may be because there are literally no results...and how is yagooglesearch supposed to know the difference between empty results and your hypothesis that it is networked related?

Your current solution to have your script handle the empty results is correct in my opinion. Your script understands the context of the network, proxies, timing, etc. much better than yagooglesearch does. Again, check out https://github.com/opsdisk/yagooglesearch#multiple-proxies

If I'm not understanding your question or situation correctly, let me know.

opsdisk commented 7 months ago

Any updates on this @photkey ?

photkey commented 7 months ago

@opsdisk Sorry, no new developments, I just solved the problem with the previously mentioned method, it's not perfect, but the script still works!

opsdisk commented 7 months ago

Great to hear, I'll close this issue out then.