search

santinic / how2

AI for the Command Line
https://how2terminal.com
MIT License
5.72k stars 156 forks source link

Google rate limit kicking in #42

Closed mttkay closed 2 months ago

mttkay commented 8 years ago

While issuing a query, the app failed with the following error:

⭐  how2 -l android launch activity in new task                                                      
-You are doing too many requests to Google. You need to wait a bit before trying again.`

If the app is using an API key shared across all users, it might be hitting a rate limit earlier than expected?

danyshaanan commented 8 years ago

I think that there isn't a shared API key. This might be caused by someone in your network running lots of google api calls. (This probably makes more sense if you are in an office rather than at home).

vasergen commented 8 years ago

Have the same problem, after playing few minutes I always got an error "You are doing too many requests to Google. You need to wait a bit before trying again"

mkohlmyr commented 8 years ago
  1. The how2 module does not use the Google Search API as it (to the best of my knowlege) has been deprecated in favour of Google Custom Search, which does not suit the needs of the project, not least of all because it imposes severe rate limits.
  2. The error actually stems from node-google, the dependency how2 uses for scraping google results, which (as it should) respects the rate limiting Google imposes on what it believes to be (correctly in this case) automated traffic. This appears to be in the neighbourhood of four queries per minute.
  3. A realistic (and useful!) fix for this issue would probably involve writing npm modules that implement the same interface as node-google (or a consistent adapter for existing modules) but for scraping bing and duck-duck-go. If searches were then scheduled round-robin from how2, it could perhaps triple the rate limit (depending on their respective limits). If there is sufficient interest in that solution, I'd be happy to take a look at it.

Hope that helps!

nicandris commented 8 years ago

I didn't even managed to use this once. I always get the rate limit msg

santinic commented 8 years ago

Yeah we need someone to write an api to scrape bing or duckduckgo

mkohlmyr commented 8 years ago

I'm happy to take a look at implementing that in the coming week, if you'd like.

santinic commented 8 years ago

@mkohlmyr awesome

mkohlmyr commented 8 years ago

Pretty sure I have ddg / bing / google basically working with an -e flag for search engine selection. If it sounds interesting I could also add support for an environment variable to set the default.

Let me know if you want any changes to https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23 before I make a PR. Forgot about the tests!

https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23 + https://github.com/mkohlmyr/how2/commit/db290793ddf01d2e6729b566a387d82445cf1f53

Let me know if you want any changes.

mkohlmyr commented 8 years ago

Ping @santinic re commits above :loudspeaker:

Happy to squash them if you feel that's cleaner, or if you want me to rename / restructure any of the changes.

santinic commented 8 years ago

@mkohlmyr sorry for the delay. It looks really good ! I have two questions:



|Cannot connect to search engine.
Error: Failed to locate valid search results on page for query: [object Object]```

But, it's amazing :) thanks for your work!
mkohlmyr commented 8 years ago

I'll hit you back up by the end of the weekend probably with some updates.

mkohlmyr commented 8 years ago

So I have done some investigating.

Your error is probably a rate limiting page that I was unaware of. It would be easy to improve the printing of the parameters, I could also create a special error message for this case by looking at the source it brings back.

FWIW it also looks like Bing has a tendency to return questions that have been closed by the user, moderator or have no answers, all of which would cause the magic search to reply with "cannot find any reasonable answer for your query". e.g. how2 reverse proxy nginx -e bing.

I have not had time to implement the changes I mentioned, but I will hopefully get to it later this week (after wednesday). If you don't want to wait, feel free to adapt the code I posted!