Add SCCACHE_CC_PREFIX for wrapping C compiler

[andreastt]

With ccache, I set the CCACHE_PREFIX output variable to icecc to have C and C++ compilation done in an icecream cluster. There should be an equivalent option to do this with sccache.

My current mozconfig for Firefox compilation looks like this:

mk_add_options 'export CCACHE_PREFIX=icecc'
mk_add_options MOZ_MAKE_FLAGS="-j100"

export CC=clang
export CXX=clang++

ac_add_options --with-ccache

Unfortunately, doing ac_add_options --with-ccache='sccache icecc' does not work because the configure script checks if 'sccache icecc' is executable.

[mbitsnbites]

The environment variable should be called SCCACHE_PREFIX to better mimic CCACHE_PREFIX, IMO.

I think it would be safe to make no assumptions about what the prefix tool does and treat it as a transparent launcher for the command line that is executed when a cache miss occurrs.

E.g. if the following call gives a cache miss:

sccache gcc -c -o hello.o hello.c

Then instead of calling gcc directly, just expand to:

$SCCACHE_PREFIX gcc -c -o hello.o hello.c

(where $SCCACHE_PREFIX is replaced by whatever the environment variable is set to, or nothing)

[mbitsnbites]

I made a quick hack that prefixes the compiler command with icecc. In gcc.rs I did:

@@ -364,7 +364,9 @@ pub fn compile<T>(creator: &T,
         Language::ObjectiveCxx => "objective-c++",
     };
-    let mut attempt = creator.clone().new_command_sync(executable);
+    let launcher = "/usr/bin/icecc";
+    let mut attempt = creator.clone().new_command_sync(launcher);
+    attempt.arg(executable);
     attempt.arg("-x").arg(language)
         .arg("-c")

...and I can see that icecc is being run. However all compilations run on my local machine so I see no speedup.

sccache does not seem to spawn more compilation processes than I have cores (I do ninja -j100). Is there some process count limit in the sccache service process?

Edit: Raising the thread count in server::start_server() from 20 to 100 does not help.

[luser]

Edit: Raising the thread count in server::start_server() from 20 to 100 does not help.

That's the only hardcoded limit in sccache, and it's not a concurrent task limit (sccache will currently run as many compile jobs as you throw at it), it's just a limit on concurrent CPU-bound tasks that we run on a background thread pool: things like hashing input files: https://github.com/mozilla/sccache/search?utf8=%E2%9C%93&q=spawn_fn&type= https://github.com/mozilla/sccache/search?utf8=%E2%9C%93&q=%22Digest%3A%3Afile%22&type=

For C compilation it's really only used for hashing the contents of the compiler binary (which is only done once per compiler path, and cached) and for writing out the compiler outputs to disk from cache hits.

There is an open PR to add jobserver support which changes things a bit, but that hasn't yet been merged.

I don't know much about how icecream works--does it have some default limit based on local core count?

[mbitsnbites]

I'm not sure about the inner workings of icecc either, but the typical scenario is that you can prefix your compilation command with icecc and start a huge bunch of those jobs in parallel (typically > 100, using something like ninja -j100).

The only way that icecc can give a performance boost is for it to run more compilation jobs (in this case, resolve more cache misses) in parallel than there are CPUs on your machine.

Not sure, but maybe there's a difference between starting these jobs from parallel threads vs from parallel processes?

[glandium]

icecream won't start 100 jobs unless it has 100 job slots available across its network.

[mbitsnbites]

Well, we have some 200 cores in our LAN, so that's not the issue. Seems like somewhere along the line there's a thread/process throttle.

All sccache clients connect to a single local sccache server, and that server will spawn compile jobs - in this case start icecc client processes. Those icecc clients will in turn synchronize with the local icecc daemon (that I believe has some kind of intelligent throttling to allow for many distributed compilations but only a few local compilations).

Seems to me that the possible points of throttling are:

1. The sccache server, that probably has some kind of job pooling.
2. The icecc client/daemon synchronization.

We will have to log and debug more to know which it is.

[luser]

There's certainly not any intentional limiting of concurrent jobs in the sccache server, but it's possible that something doesn't work as expected in this scenario.

[julienw]

Is it possible to make this set-up work by simply changing the PATH? On my system icecc gets installed with symbolic links to gcc g++ c++ and cc in /usr/lib/icecc/bin/, so I add export PATH=/usr/lib/icecc/bin:$PATH to mozconfig. This just works with ccache (without CCACHE_PREFIX) but not sccache.

[zougloub]

Adding that SCCACHE_PREFIX optional feature would be very nice, thank you.