search

modrzew / pokeminer

Pokemon location scraper
MIT License
220 stars 62 forks source link

Start scan from current point after restart, soft-ban detection, and a shutdown function on banned #249

Open gunawanputra opened 7 years ago

gunawanputra commented 7 years ago

Soft-ban could be detected when get_map_objects response only have current_timestamp and cell_id, which will make a point processed without any pokemon and fort found. By counting how many times forts result is empty, we could find out when a worker is banned. Right now worker thread will be terminated (I use word shutdown) when it get banned.

MalformedResponse exception currently is not in use. I put a try-except block when assign value of response_dict to map_objects and handle the TypeError exception.

I did this because response_dict have a NoneType when "server is busy or offline" and in this situation I think it's better if worker simply move to the next point and try to send another request rather than restart itself. There is also "sequence index must be integer, not 'str'" error, happen when worker considered not logged in by pgoapi. Worker need an immediate restart and relogin on this error.

For now I just put banned_count in the try-except block to shutdown the worker if it keep getting TypeError. It's better to create a counter and count how many times this TypeError happen, especially sequence index error, and raise MalformedResponse to restart worker when it reach the limit.

--edit-- On pgoapi commit 8c1c176, api will raise NotLoggedInException instead of return NotLoggedInException. This will make "TypeError : sequence index must be integer, not 'str'" exception never happen again. NotLoggedInException exception handling added in try-except block in cycle while loop, will restart worker when this exception raised.

banned_count removed from TypeError exception handling.

Aiyubi commented 7 years ago

This really is too many Features in a single PR. Especially since ban detection is already in master.

gunawanputra commented 7 years ago

Ban detection in master only detect permanent ban (status_code=3), and will keep restart worker when BannedAccount exception raised.

fxbt commented 7 years ago

Workers are restarting because we experienced some false positive with status_code 3. You can look at the discussion here : #244

gunawanputra commented 7 years ago

Restarting because of some false positive with status_code 3, but then workers with permanent banned account will just keep restarting. On the newest commit, I even add permanent ban boolean status that set to true when response have status_code 3. Then if it's just false positive like what you said, will set to false after point is processed.