icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.61k stars 252 forks source link

Provide a make compatible jobserver #635

Open Spudd86 opened 2 months ago

Spudd86 commented 2 months ago

Provide a make jobserver that will control the number of active jobs. It should be able to use slots in the local iceccd even for jobs icecream can't distribute. This way the real number of active jobs can be managed for the local system.

It should also move around job slots when for example ccache is in use, ccache should take a local job slot until it runs icecc, which should take over the job slot and transfer it according to the scheduler, then delay returning the result to ccache until a local slot opens up for it to occupy.

Basically the idea is that icecc gives up it's local slot when it runs a remote command and before exiting acquires a new one. There would need to be some sort of wrapper around build invocation that sets up the jobserver and sets another environment variable that can be used by icecc to talk to the jobserver outside the make jobserver protocol.

This would enable make based builds to be limited properly without modifying the actual build rules to use icerun and would ensure that the system never exceeds the number of active local jobs that iceccd provides.