search

openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
462 stars 74 forks source link

Sidekiq processes using 100% CPU #997

Open henare opened 8 years ago

henare commented 8 years ago

This is causing the queue to build up because the workers take forever to finish running scrapers. Right now there's 63 scrapers queued but only 2 containers actually running.

Extracted from #992.

henare commented 8 years ago

I found some duplicate jobs on the queue. After clearing them out there's a lot more running containers.

henare commented 8 years ago

So the duplicates definitely make the sidekiq process run glacially but I've also found a bunch of new scrapers that are in infinite loops and logging output for every loop too. That's bound to peg the CPU but what should we do in that case? I think the only solution would be to add some kind of CPU time restriction. It would be a shame to add another constraint.