API seems to go on and offline when hit with heavy load

harkib commented 4 years ago

Over the last few days I observed the following.

April 11th: working all day, only doing small non consecutive searches April 12th:

working with small non consecutive searches
(8pm ish) worked, hit API with consecutive (30 threads) searches (~60 searches, ~3000 links)
(9pm ish) returns nothing for simple test case
(11pm ish) returns results for simple test
(11pm+ ish) worked, hit API with consecutive (30 threads) searches (~60 searches, ~3000 links)
(11pm++ ish) returns nothing for simple test case

April 13th:

(9pm) worked, hit API with non consecutive (1 thread, processing time after each search) searches (~60 searches, ~5000 links)
(right after large search ~11:40pm) returns results for simple test
(11:50pm) attempted to hit the API with the same searches as 9pm but much less processing time between searches. Finished 8 searches before returning nothing for remaining searches

April 14th: (noon): working, simple test

Are there some limits or guidelines we should follow while this project is under development?

mjlabe commented 4 years ago

If you look at the closed PRs, this could be any number of reasons. Most services will block you if you scrape too often or in an automated fashion.

harkib commented 4 years ago

Seems to be a result of google blocking IP

Iceloof / GoogleNews