Have master spawn Spool workers as needed

[funkybob]

Is there any reason the master task can't monitor the spool dir, and when there are jobs it spawn spool workers up to the limit.

This would avoid the spool workers occupying memory when not needed, and also have a single, serialised point for handing out spool jobs.

Clearly the downside is slower start up on jobs, but given the default polling period is 30 seconds, this will only be an issue when spawn cost is great due to yawn-and-stretch costs.

[unbit]

Prototype of --spooler-cheap:

• the master monitors each spooldir
• every T (spooler-cycle) it checks for spoolfiles in each spooldir
• if a non-locked spoolfile is found (locked file means a spooler is alredy processing it) and the number of spooler processes for that spooldir is lower than --spooler-processes, spawn a new spooler
• the new spooler runs the tasks until there are no more tasks to run, then it dies

--spooler-cheap --spooler /foo --spooler /bar --spooler-processes 4

will start with 0 spoolers, and will rise up to 8 (4 for /foo, 4 for /bar)

[unbit]

Pushed an implementation in 2.1, it is not invasive so we could backport it to 2.0 too.

Let me know how it works for you

[ultrabug]

Very handy implementation, will give it a try and report !

[xrmx]

So it looks like there's no request to have it backported, closing then. Feel free to reopen.

[oliveiraev]

Could you please backport it to 2.0 branch? I've tried but failed :disappointed: