search

paultag / moxie

moxie
MIT License
28 stars 7 forks source link

concurent start limit #54

Open paultag opened 9 years ago

paultag commented 9 years ago

rate limit the number of jobs being brought up at once. Let's do 2 at a time at most?

mileswwatkins commented 9 years ago

This would be lovely; I'm concerned about the effect of every Open State in Eastern Time starting their scrapers at the same time, haha. Could tweak the moxie-jobs start time individually, but this seems like a much better solution.

mileswwatkins commented 9 years ago

Yeah, with so many jobs starting promptly at local midnight last night, we had tons of MIAs. @rshorey and I are going to try to knock this out.

paultag commented 9 years ago

Try using a blocking channel to send jobs to a central worker, and use that for controlling concurent bringup :+1:

On Tue, Aug 18, 2015 at 4:17 PM, Miles Watkins notifications@github.com wrote:

Yeah, with so many jobs starting promptly at local midnight last night, we had tons of MIAs. @rshorey https://github.com/rshorey and I are going to try to knock this out.

— Reply to this email directly or view it on GitHub https://github.com/paultag/moxie/issues/54#issuecomment-132225925.

:wq