Sccache is not working after resume on macOS-12

linux4life798 commented 10 months ago

On macOS-12, sccache is not working after cache resuming and then fails to post stats during the cleanup. The latter issue causes the new cache to not be saved.

Checkout the following consecutive builds:

Build Job 48

This started with no prior cache and ran for 2h 7m 38s.

Post job cleanup.
sccache stats
  /usr/local/bin/sccache -s
  Compile requests                   4917
  Compile requests executed          4917
  Cache hits                            1
  Cache hits (C/C++)                    1
  Cache misses                       4916
  Cache misses (C/C++)               4916
  Cache timeouts                        0
  Cache read errors                     0
  Forced recaches                       0
  Cache write errors                    0
  Compilation failures                  0
  Cache errors                          0
  Non-cacheable compilations            0
  Non-cacheable calls                   0
  Non-compilation calls                 0
  Unsupported compiler calls            0
  Average cache write               0.001 s
  Average compiler                  6.422 s
  Average cache read hit            0.000 s
  Failed distributed compilations       0
  Cache location                  Local disk: "/Users/runner/work/llvm-project-ci-test/llvm-project-ci-test/.sccache"
  Use direct/preprocessor mode?   yes
  Version (client)                0.7.4
  Cache size                          141 MiB
  Max cache size                      500 MiB
Save cache using key "sccache-macOS-12-2024-01-06T20:34:28.719Z".
/usr/local/bin/gtar --posix -cf cache.tzst --exclude cache.tzst -P -C /Users/runner/work/llvm-project-ci-test/llvm-project-ci-test --files-from manifest.txt --delay-directory-restore --use-compress-program zstdmt
Cache Size: ~155 MB (162697604 B)
Cache saved successfully

In between Jobs Canceled Without Starting

Build Job 50

This build started with Job 48's cache and ran for 2h 4m 11s.

...
Install sccache
  /bin/sh -xc brew install sccache
  + brew install sccache
  ==> Downloading https://ghcr.io/v2/homebrew/core/sccache/manifests/0.7.4
  ==> Fetching sccache
  ==> Downloading https://ghcr.io/v2/homebrew/core/sccache/blobs/sha256:2398c0375c5c282810ff42b90d44dbdf56fb955c0c7ccfc980fd5bb2c0c8fbb4
  ==> Pouring sccache--0.7.4.monterey.bottle.tar.gz
  🍺  /usr/local/Cellar/sccache/0.7.4: 7 files, 15.1MB

Restore cache
  Received 0 of 162697604 (0.0%), 0.0 MBs/sec
  Received 33554432 of 162697604 (20.6%), 16.0 MBs/sec
  Received 67108864 of 162697604 (41.2%), 21.0 MBs/sec
  Received 96468992 of 162697604 (59.3%), 22.7 MBs/sec
  Received 109051904 of 162697604 (67.0%), 20.5 MBs/sec
  Received 138412032 of 162697604 (85.1%), 21.5 MBs/sec
  Cache Size: ~155 MB (162697604 B)
  /usr/local/bin/gtar -xf /Users/runner/work/_temp/b93e6650-5c21-4407-adb6-9f6730c563af/cache.tzst -P -C /Users/runner/work/llvm-project-ci-test/llvm-project-ci-test --delay-directory-restore --use-compress-program unzstd
  Received 162697604 of 162697604 (100.0%), 21.7 MBs/sec
  Cache restored successfully
  Restored from cache key "sccache-macOS-12-2024-01-06T20:34:28.719Z".
...

Post job cleanup.
sccache stats
  /usr/local/bin/sccache -s
  sccache: error: failed to get stats from server
  sccache: caused by: Failed to send data to or receive data from server. Mismatch of client/server versions?
  sccache: caused by: Failed to read response header
  sccache: caused by: failed to fill whole buffer
  Warning: Saving cache failed: Error: The process '/usr/local/bin/sccache' failed with exit code [2](https://github.com/linux4life798/llvm-project-ci-test/actions/runs/7433668388/job/20227285351#step:14:2)

In between Jobs Canceled Without Starting

Build Job 53

This build started with Job 48's cache, also, and ran for 2h 5m 4s.

...
Install sccache
  /bin/sh -xc brew install sccache
  + brew install sccache
  ==> Downloading https://ghcr.io/v2/homebrew/core/sccache/manifests/0.7.4
  ==> Fetching sccache
  ==> Downloading https://ghcr.io/v2/homebrew/core/sccache/blobs/sha256:2398c0375c5c282810ff42b90d44dbdf56fb955c0c7ccfc980fd5bb2c0c8fbb4
  ==> Pouring sccache--0.7.4.monterey.bottle.tar.gz
  🍺  /usr/local/Cellar/sccache/0.7.4: 7 files, 15.1MB

Restore cache
  Received 0 of 162697604 (0.0%), 0.0 MBs/sec
  Received 33554432 of 162697604 (20.6%), 16.0 MBs/sec
  Received 58720256 of 162697604 (36.1%), 18.7 MBs/sec
  Received 79691776 of 162697604 (49.0%), 19.0 MBs/sec
  Received 109051904 of 162697604 (67.0%), 20.8 MBs/sec
  Received 134217728 of 162697604 (82.5%), 21.3 MBs/sec
  Cache Size: ~155 MB (162697604 B)
  /usr/local/bin/gtar -xf /Users/runner/work/_temp/bb8d26a4-00a2-4c35-84f2-1e06d6a9322d/cache.tzst -P -C /Users/runner/work/llvm-project-ci-test/llvm-project-ci-test --delay-directory-restore --use-compress-program unzstd
  Received 162697604 of 162697604 (100.0%), 22.1 MBs/sec
  Cache restored successfully
  Restored from cache key "sccache-macOS-12-2024-01-06T20:34:28.719Z".
...

Post job cleanup.
sccache stats
  /usr/local/bin/sccache -s
  sccache: error: failed to get stats from server
  sccache: caused by: Failed to send data to or receive data from server. Mismatch of client/server versions?
  sccache: caused by: Failed to read response header
  sccache: caused by: failed to fill whole buffer
  Warning: Saving cache failed: Error: The process '/usr/local/bin/sccache' failed with exit code [2](https://github.com/linux4life798/llvm-project-ci-test/actions/runs/7434184703/job/20228420650#step:14:2)

These and more recent builds can be seen here.

hendrikmuhs commented 10 months ago

@linux4life798

Am I getting this right:

this is caused by an upgrade of the macos runner from macos-11 to macos-12?
the version of sccache hasn't changed
sscache is installed via brew

So the issue seems to be with the sccache binary on the newer macos? Is it the same binary that gets installed on macos-11 and macos-12?

This doesn't seem to be an issue with the action, however it is good to keep this issue as a warning for others.

linux4life798 commented 10 months ago

this is caused by an upgrade of the macos runner from macos-11 to macos-12?

@hendrikmuhs, yes, I bumped the runner version from macOS-11 to macOS-12, but the upgrade itself doesn't seem to be the issue. The different runners are configured to use different GitHub caches, which is fine. I tried to show in the above examples that the first builder on macOS-12 starts with no cache and then restores that cache on the next build.

So the issue seems to be with the sccache binary on the newer macos?

Very possible.

Is it the same binary that gets installed on macos-11 and macos-12?

It looks like brew is installing the same version, 0.7.4, on macOS-11 as it does on macOS-12, but it takes massively more time to install on macOS-11 (like 20 mins vs 20 seconds). See https://github.com/linux4life798/llvm-project-ci-test/actions/runs/7522258649/job/20474458939.

linux4life798 commented 10 months ago

I was concerned that the restore and save actions might be somehow invoking different versions of sccache, like the error says:

sccache: caused by: Failed to send data to or receive data from server. Mismatch of client/server versions?

Or, maybe the save action is being invoked too close to the last compilation, causing corruption?

hendrikmuhs / ccache-action