search

michaelforney / samurai

ninja-compatible build tool written in C
Other
832 stars 47 forks source link

Support make jobserver protocol #71

Open rossburton opened 3 years ago

rossburton commented 3 years ago

GNU Make has a neat feature called the jobserver protocol, where the top-level Make can allocate a specific number of job slots, and child makes can take slots to do work in. This was designed to stop the parallelisation problem where a top-level make -j10 may potentially spawn 10 separate sub-makes all with -j10 so there's now 100 parallel jobs.

However, it's also useful for resource control is systems which built multiple pieces of software at once. For example, Bitbake can build N different pieces of software at once, and each of those is passed a -jM flag. If each of these N tasks is compiling then thats's N*M jobs so you don't want N or M to be too high, but if only 1 of N is building then you want M to be high.

With the job server protocol there are N slots in total for all sub makes, so you can control the resource utilisation more accurately. If Samurai also supported the jobserver protocol instead of just -j then it could join in the resource pooling and builds can be more efficient.

The canonical implementation of jobserver is in the GNU Make sources. The documentation is pretty vague as it is considered an implementation detail, but it hasn't changed in 20 years... The prototype BitBake support for a master jobserver queue is at http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237

rossburton commented 3 years ago

cc @rpurdie

bonzini commented 2 years ago

Hi, I have done a quick and dirty implementation (less than 200 lines of code) but it uses pthreads. Should I submit a pull request or should I first try to rework it without threads?

michaelforney commented 2 years ago

@bonzini, I am interested in taking a look at what you've got, but I'd definitely prefer an implementation without threads.

eli-schwartz commented 2 years ago

BTW since https://git.savannah.gnu.org/cgit/make.git/commit/?id=7ad2593b2d2bb5b9332f4444d8bf93ac6f958bc6 GNU make supports named pipes as an alternative to anonymous pipes.

Once it actually gets into a release that should be pretty nifty.

mid-kid commented 1 year ago

I feel it's worth noting that gcc picks up on this jobserver when using gcc -flto for LTO builds, and as such is very useful for speeding that up. Using -flto=n to specify the amount of LTO linking jobs isn't great when it comes to projects linking multiple programs concurrently. Oh, and -flto=n still prefers using a jobserver if available.

bonzini commented 6 months ago

@bonzini, I am interested in taking a look at what you've got, but I'd definitely prefer an implementation without threads.

I opened a new PR in #104 now that someone reminded me of it. It does use threads, which is almost mandatory because the GNU Make protocol requires a blocking file descriptor. But the implementation is quite nifty if you ask me. :)