Linux -> Mac & enableCrossOsArchive -> Cache not found for input keys

[niall-byrne]

When creating a simple cache of downloaded content in my Linux steps, and caching it, each Linux machine has no problem mounting the created cache and reusing the content.

When the workflow gets to the OSX machine, it does not register a cache hit, and instead creates a new cache instance with the identical key.

Config:

Linux Machines (happy to use the cache):

   - name: Ansible Lint -- Mount Ansible Cache
      uses: actions/cache@v3
      with:
        enableCrossOsArchive: true
        key: ansible-${{ hashFiles('./template/{{cookiecutter.profile_slug}}/profile/requirements.yml') }}-${{ env.CACHE_TTL }}
        path: ansible_cache

OSX Machine (creates it's own cache):

    - name: Apply Profile -- Mount Ansible Cache
       uses: actions/cache@v3
       with:
         enableCrossOsArchive: true
         key: "ansible-${{ hashFiles('./template/{{cookiecutter.profile_slug}}/profile/requirements.yml') }}-${{ env.CACHE_TTL }}"
          path: ansible_cache

Am I assuming functionality that does not exist?

Example Workflow Run: https://github.com/osx-provisioner/profile-generator/actions/runs/4197900167/jobs/7280858264

[kleisauke]

I'm experiencing the same issue. See for example this debug CI run: https://github.com/kleisauke/wasm-vips/actions/runs/4197093611/jobs/7279827475#step:4:42

Reverse engineering these version keys yields to:

$ node -p "require('crypto').createHash('sha256').update('lib|gzip|1.0').digest('hex')"
2b8f91fb12183714d33b0e7ff6189a3363063c17729d1c6529f02e1fe17e91d8
$ node -p "require('crypto').createHash('sha256').update('lib|zstd-without-long|1.0').digest('hex')"
b8f3c3424597ca704ee437c4e20b46e1ec8212372b530f2cc52240b6d527d6c9

So, it looks like it's trying to use restore the cache using CompressionMethod.Gzip, which is inappropriate since the cache was saved with CompressionMethod.ZstdWithoutLong.

I suspect this is being caused by the zstd version bump to 1.5.4, which was recently rolled out to GitHub actions. See these commits: https://github.com/actions/runner-images/commit/bf50e0da6fd3107aabdf950a2fae72a98aaf939b https://github.com/actions/runner-images/commit/6b424ca736d5a5d9929beba8514b2c3dbaf0bbb6

Probably this is causing zstd --version no longer to output zstd command line interface, making CompressionMethod default to CompressionMethod.Gzip, which breaks the enableCrossOsArchive: true option. https://github.com/actions/toolkit/blob/d2b7d85e7cb61467395604aae1b84439c320cb84/packages/cache/src/internal/cacheUtils.ts#L100

[kleisauke]

... commit https://github.com/facebook/zstd/commit/9c93dd71cdb301595a8a9e364348967a173c010c seems to be the culprit (released in zstd 1.5.4)

[parched]

I've added this step to my workflow as a temporary workaround

    # zstd 1.5.4 broke actions/toolkit https://github.com/actions/toolkit/issues/1341
    - if: runner.os == 'Windows'
      run: |
        if zstd --version | grep -q '1.5.[4-9]'; then
          choco install zstandard --version 1.5.0
          echo 'C:\ProgramData\chocolatey\bin'>>"$GITHUB_PATH"
        fi
      shell: bash

[panva]

enableCrossOsArchive is only documented to have an effect on windows runners. What I'm seeing now is that macOS gets cache-misses on linux-created caches.

[parched]

@panva yes, it's not related to OS, just the version of Zstandard that is installed. It just happens the Windows and macOS github hosted runners have Zstandard 1.5.4 installed and the Linux ones don't yet.

[kleisauke]

Here's a temporary workaround for both Windows and macOS:

      # https://github.com/actions/toolkit/issues/1341
      - name: Downgrade zstd on Windows
        if: runner.os == 'Windows'
        run: |
          Invoke-WebRequest https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-v1.5.2-win64.zip -OutFile $env:RUNNER_TEMP\zstd.zip
          Expand-Archive $env:RUNNER_TEMP\zstd.zip -DestinationPath $env:RUNNER_TEMP
          Add-Content $env:GITHUB_PATH "$env:RUNNER_TEMP\zstd-v1.5.2-win64"
      - name: Downgrade zstd on macOS
        if: runner.os == 'macOS'
        run: |
          brew uninstall --ignore-dependencies zstd
          git -C "$(brew --repo homebrew/core)" checkout d3f04bd Formula/zstd.rb
          brew install zstd

(see e.g. commit https://github.com/kleisauke/wasm-vips/commit/d475fb3b61f5b458fc444943dd035c1b42827f75)

[marcdragon123]

another temporary workaround is:

cd “$(brew —repo homebrew/core)” && git checkout d3f04bddb113beb03690732a13ea8bc9af33ceaf
HOMEBREW_NO_UPDATE=1 brew unlink zstd
HOMEBREW_NO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 brew install zstd

This will unlink zstd 1.5.4 and install 1.5.2. Since 1.5.2 was the last functional version.

[niall-byrne]

@panva you're right, but when I saw identical cache keys I wanted to try all the options before filing an issue...

[niall-byrne]

This appears to be resolved now on builds I'm seeing.

[narumi147]

yesterday linux also updated zstd to 1.5.4, and then cache restore failed on linux the same os.

[cdce8p]

I've opened https://github.com/actions/toolkit/pull/1345 with a possible fix for the version parser.

[niall-byrne]

I understand, thanks.

[panva]

Issue seems to have resolved for my workflows.

[niall-byrne]

The linux upgrade took care of most of it for me too, the caches are now created with the same method on OSX and Linux.
I am assuming that using the new version is the path forward eventually.

I'm hesitant to close this, until the issue with version parsing is resolved however.

[cdce8p]

The linux upgrade took care of most of it for me too, the caches are now created with the same method on OSX and Linux. I am assuming that using the new version is the path forward eventually.

I'm hesitant to close this, until the issue with version parsing is resolved however.

It might work now, however as the version parsing is still broken, the actions defaults back to using glib instead of zstd. Cache save times are about 3x slower because of that for what I've seen.

[Juni66623]

J

[parched]

It looks like most (all?) of the GitHub hosted runners have updated to zstd 1.5.4, so they're working together by falling back to gzip instead. However, we have some self hosted runners still on zstd 1.5.0, so now I'm using this workaround on Linux too.

    - if: runner.os == 'Linux'
      run: |
        if zstd --version | grep -q 'Zstandard CLI'; then
          printf '#!/bin/sh\n'"$(which zstd)"' "$@" | sed "s/Zstandard CLI/zstd command line interface/g"' > "$RUNNER_TEMP/zstd"
          chmod +x "$RUNNER_TEMP/zstd"
          echo "$RUNNER_TEMP">>"$GITHUB_PATH"
        fi
      shell: bash

[rohanKanojia]

Hi guys, I'm facing the same issue. Were you able to find some workaround for this? I tried reverting actions/cache to previous version but it doesn't seem to fix the issue.

[DerTim1]

@rohanKanojia That won't be helpful, due to the fact, that the hosted runner needs to get downgraded in the version of zstd until the issue is fixed here.

You can do this on hosted runner (Ubuntu):

- name: Install Zstd
  run: sudo apt-get update && sudo apt-get install zstd=1.4* && sudo rm -rf /usr/local/bin/zstd

I only tested it on ubuntu-20.04, but ubuntu-latest (22.04) should also have an older zstd version.

The issue need to be fixed here: https://raw.githubusercontent.com/actions/cache/main/dist/restore-only/index.js Line 1182:

function getCompressionMethod() {
    return __awaiter(this, void 0, void 0, function* () {
        const versionOutput = yield getVersion('zstd');
        const version = semver.clean(versionOutput);
        if (!versionOutput.toLowerCase().includes('zstd command line interface')) {

@kleisauke mentioned the corresponding commit on zstd, thanks for that!