search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.85k stars 552 forks source link

Does sccache use the local cores during the distributed build? #1071

Open bliednov opened 2 years ago

bliednov commented 2 years ago

Maybe such a question was already answered somewhere, but I could not find it anywhere.

Env:

  1. Scheduler machine with 1 core - sccache-dist scheduler is running.
  2. Builder machine with 32 cores - sccache-dist server is running.
  3. Client machine with 32 cores - sccache is configured to use the distributed build.

The question: If I run to build on the client machine make -j64, will it also utilize the local cores? It seems to me that it uses them only when it falls back to local in case of an error.

cr1901 commented 2 years ago

I also have the same question (without having the ability to set up a working distributed compile right now). I see multiple answers from various issues, and am detailing what I found for my own reference.

TLDR

A single machine can be a build server that does compiling, a client w/ compile jobs that delegates to a build server, or a client that compiles all jobs locally. Pick one. But see "Workaround?"

Longer Version

I would appreciate clarification where I'm wrong:

  1. sccache has a concept of a client (sccache), local server (sccache --start-server), build server (sccache-dist), and scheduler (also part of the sccache-dist binary, but it has different responsibilities from each build server and should only be run on one machine).
    • The client wraps the compiler and talks to the remaining parts above with the goal of retrieving/writing out a compiled artifact.
    • The local server handles using the cores on your machine to create compile artifacts that sccache stores somewhere (possibly storing the cache artifacts remotely, unrelated to the build server responsibilities). All clients spawn a local server.
    • Build servers are dedicated machines for accepting jobs remotely w/ coordination from a scheduler.
  2. sccache will not use cores on your current machine as part of delegating work from the scheduler. Only dedicated build servers are considered: https://github.com/mozilla/sccache/issues/617
  3. sccache will spawn a local server for each client when doing local compilation to cache artifacts. Related to 2., there are no provisions to reuse this local server to accept jobs on behalf of a client on another machine: https://github.com/mozilla/sccache/issues/620
  4. If the scheduler says no jobs are available on remote machines, sccache will fall back to the local server for local compilation: https://github.com/mozilla/sccache/issues/708.
    • I have never seen the "falling back" message compiling only locally, so I guess sccache knows how to distinguish "we should only spawn a local server" from "we should try to communicate with a scheduler" somehow. Maybe the client config file?

Workaround?

Is it possible to fool a client that's trying to delegate jobs to a build server to use local cores by also running a build server on your local machine?

mitchhentges commented 2 years ago

Decent write-up, it all aligns with my understanding of behaviour. I'm not aware of a workaround to leverage the cores of a client in addition to distributed workers during a build - I think that kind of work is blocked on #617.