search

modrzew / pokeminer

Pokemon location scraper
MIT License
220 stars 61 forks source link

Workers being killed #101

Open ghost opened 8 years ago

ghost commented 8 years ago

I have many workers being killed and I am only at ~80 points per cycle. Running 1226 threads across 810km utilizing 130% cpu on a 6vCore server with 8GB RAM .. do I need to run less workers? It looks like each core is only at ~21% load but many workers are staying at 0%. Also how much RAM usage are you guys seeing with high amounts of workers?

amistad18 commented 8 years ago

I didn't have any problem with RAM usage (nearly 500MB for ~200 workers), but because of CPU usage I had to decrease number of workers on my VPS instance. But I figure out a temporary solution, it probably won't help you with workers being killed (anything interesting in log file?), but it may decrease CPU usage for you - https://github.com/modrzew/pokeminer/pull/100

ghost commented 8 years ago

nothing that I can see in the log file.. it all looks pretty normal. The area I am scanning is all metropolitan, disabling workers wont help.. but I am sure I will find this handy when I move on to the lakes and rivers.. thanks!

Aiyubi commented 8 years ago

are you using the sqlite file? Might be that your database is slowing everything down

SapitoSucio commented 8 years ago

I have 66 workers at 25 points per cycle, when it starts it's using about 10~15% CPU (Web.py & Worker.py) but after a several hours later, the CPU goes upto 60% and I need to stop them and even sometimes restart my PC because it's so laggy

LordLioath commented 8 years ago

This sounds like your using too many workers for one instance of PokeMiner

You will want to limit it to about 200-250 workers per running instance, if you need to use more, you may want to split the area between two copies of PokeMiner with their own configs, thats what I do currently and I'm running about 10 instances of 200 over most of the key areas of Florida.

subzerofun commented 8 years ago

@thappy 1226 workers?? are you scanning area 51? how the hell is it even possible to run that many workers without getting banned? didn´t mean that as an offense - but that´s really a lot! do all the worker requests come from the same IP?

modrzew commented 8 years ago

And I thought 800 workers someone reported a few weeks ago was overkill.

All of you may be interested in what I've been doing in coroutines branch. I'm testing it locally right now, but from what I see it has reduced CPU usage due to lower number of threads, lower number of database connections, and various stability fixes.