search

ArchiveTeam / NewsGrabber

Grabbing all news.
62 stars 32 forks source link

rsync concurrency limit #74

Closed HarryC145 closed 7 years ago

HarryC145 commented 8 years ago

We need a way to limit the rsync concurrency on the workers -> master server. At the moment, the master server has close to 1000 open rsync processes and is struggling with it

Arkiver2 commented 7 years ago

Implemented with new grabber script.