Use Thread/Process pools

ArchiveTeam / NewsGrabber

Grabbing all news.

62 stars 32 forks source link

Use Thread/Process pools #61

Open PromyLOPh opened 8 years ago

PromyLOPh commented 8 years ago

Using thread or process pools and queues instead of starting a new thread one for each grab (or subtask) would limit the number of jobs running at the same time and thus keep the server load at manageable levels.

HarryC145 commented 8 years ago

load average: 150.87, 129.18, 115.0