I need to clean this up a bit, but would love early review.
@toddr and @atoomic This is based off the updated PR for too many open handles. I was hoping you could pull this down and give it a try, maybe set --max-open-jobs to a lowish number like 100, and then also add --spawn-worker-at-max to see if that solves your problem without the collector(s) getting horribly backed-up. If you are in a situation where the collector is THAT backed up then your tests may finish well before yath does which would cost you extra time, assuming your max open handles limit is per-process and not per-user extra collectors can help work though the backlog.
This method does not limit the number of collectors, rather whenever a collector hits the max open files it forks, the fork finishes up all the jobs already started leaving the main collector to grab new jobs, once all the jobs of the forked collector finishes it exits, if another backup happens a new one forks. If you have multiple backups you get multiple collectors. So for example setting -j100, and setting --max-open-jobs to 1 would mean 100 collectors get spawned (yikes! do not do that!) it could actually climb higher than 100 if the collectors take too long even on a single job. So you really do want a decently high --max-open-jobs.
Like all things, you need to balance the 3 collector settings and tune them to something that works right for your setup, which is why I wanted all these to be configurable.
I need to clean this up a bit, but would love early review.
@toddr and @atoomic This is based off the updated PR for too many open handles. I was hoping you could pull this down and give it a try, maybe set
--max-open-jobsto a lowish number like 100, and then also add--spawn-worker-at-maxto see if that solves your problem without the collector(s) getting horribly backed-up. If you are in a situation where the collector is THAT backed up then your tests may finish well before yath does which would cost you extra time, assuming your max open handles limit is per-process and not per-user extra collectors can help work though the backlog.This method does not limit the number of collectors, rather whenever a collector hits the max open files it forks, the fork finishes up all the jobs already started leaving the main collector to grab new jobs, once all the jobs of the forked collector finishes it exits, if another backup happens a new one forks. If you have multiple backups you get multiple collectors. So for example setting -j100, and setting --max-open-jobs to 1 would mean 100 collectors get spawned (yikes! do not do that!) it could actually climb higher than 100 if the collectors take too long even on a single job. So you really do want a decently high --max-open-jobs.
Like all things, you need to balance the 3 collector settings and tune them to something that works right for your setup, which is why I wanted all these to be configurable.