I am trying to get a list of apps, which is in large scale. My scritp is to get 50 apps of each category in a time to the total number 2000.
The problem is that as the offset in the search is larger than 500. The search will come across a failure. I do not think it is caused by the blocking of my IP address. In my manual test, the successful search (offset less than 500) follows tightly with the search with failure. Also, in my script, it sleeps for 10 mins between two searchs.
I appreciate your efforts on reading this problem.
Hello,
Thank you very much for the good work first.
I am trying to get a list of apps, which is in large scale. My scritp is to get 50 apps of each category in a time to the total number 2000.
The problem is that as the offset in the search is larger than 500. The search will come across a failure. I do not think it is caused by the blocking of my IP address. In my manual test, the successful search (offset less than 500) follows tightly with the search with failure. Also, in my script, it sleeps for 10 mins between two searchs.
I appreciate your efforts on reading this problem.
Best regards, Zhengyang
java -jar googleplaycrawler-0.3.jar -f crawler.conf list BOOKS_AND_REFERENCE -s apps_topselling_free -n 50 -o 200 [success]
java -jar googleplaycrawler-0.3.jar -f crawler.conf list BOOKS_AND_REFERENCE -s apps_topselling_free -n 50 -o 500 [failure]