Cache save failure

[mkolopanis]

Hi there, not entirely sure if this question is best suited here or on the cache action, but we have been trying to utilize the benchmarking caching as described in this repository with a few issues. I work with a group of developers looking to utilize the benchmark.

I've attached a copy of our action yaml below, but we are using the split restore/save parts of the cache action because we only want to save benchmarks run against our main branch. However any subsequent benchmarks have started yielding failures as follows.

Run actions/cache/save@v4
  with:
    path: ./cache
    key: macos-latest-benchmark
    enableCrossOsArchive: false
  env:
    PYTHON: 3.1[2](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions/runs/11281421062/job/31377142366#step:9:2)
    INPUT_RUN_POST: false
    CONDA: /Users/runner/miniconda[3](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions/runs/11281421062/job/31377142366#step:9:3)
    CONDA_PKGS_DIR: /Users/runner/conda_pkgs_dir
/opt/homebrew/bin/gtar --posix -cf cache.tzst --exclude cache.tzst -P -C /Users/runner/work/pyuvdata/pyuvdata --files-from manifest.txt --delay-directory-restore --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key macos-latest-benchmark, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/main, Key: macos-latest-benchmark, Version: e[4](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions/runs/11281421062/job/31377142366#step:9:4)4d85be67565de5351873d607403f9baacc7ff08f9c4d36df4618c8a[5](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions/runs/11281421062/job/31377142366#step:9:5)061620
Warning: Cache save failed.

This warning lead us to the cache action's readme stating all caches are immutable but with some workaround. We are wondering if there is a better method known to the benchmark team to update the cache when we run benchmarking on main, or if using the "restore-key" method mentioned in the linked document is the only way.

  benchmark:
    name: Performance Benchmark
    needs: tests
    env:
      PYTHON: ${{ matrix.python-version }}
    runs-on: ${{ matrix.os }}
    defaults:
     run:
       # Adding -l {0} helps ensure conda can be found properly in windows.
       shell: bash -l {0}
    strategy:
      fail-fast: false
      matrix:
        os: [macos-latest, windows-latest, ubuntu-latest]
        python-version: ["3.12"]
        include:
          - env_name: pyuvdata_tests_windows
            os: windows-latest
          - env_name: pyuvdata_tests_mac_arm
            os: macos-latest
          - env_name: pyuvdata_tests
            os: macos-13
            python-version: "3.12"
          - env_name: pyuvdata_tests
            os: ubuntu-latest
    steps:
      - uses: actions/checkout@main
        with:
          fetch-depth: 0

      - name: Setup Miniforge
        uses: conda-incubator/setup-miniconda@v3
        with:
          miniforge-version: latest
          python-version: ${{ matrix.python-version }}
          environment-file: ci/${{ matrix.env_name }}.yml
          activate-environment: ${{ matrix.env_name }}
          run-post: false

      - name: Conda Info
        run: |
          conda info -a
          conda list
          PYVER=`python -c "import sys; print('{:d}.{:d}'.format(sys.version_info.major, sys.version_info.minor))"`
          if [[ $PYVER != $PYTHON ]]; then
            exit 1;
          fi

      # also install benchmark utility
      - name: Install
        run: |
          pip install pytest-benchmark
          pip install .

      - name: Run benchmark
        run: |
          pytest --benchmark-only --benchmark-json output.json

      # Download previous benchmark result from cache (if exists)
      - name: Download previous benchmark data
        uses: actions/cache/restore@v4
        with:
          path: ./cache
          key: ${{ matrix.os }}-benchmark

      # Run `github-action-benchmark` action
      # this step also EDITS the ./cache/benchmark-data.json file
      # We do not need to add output.json to the cache directory
      - name: Compare benchmarks
        uses: benchmark-action/github-action-benchmark@v1
        with:
          # What benchmark tool the output.txt came from
          tool: 'pytest'
          # Where the output from the benchmark tool is stored
          output-file-path: output.json
          # Where the previous data file is stored
          external-data-json-path: ./cache/benchmark-data.json
          # Workflow will fail when an alert happens
          fail-on-alert: true
          # Comment on the PR if the branch is not a fork
          comment-on-alert: true
          # Enable Job Summary for PRs
          summary-always: true
          github-token: ${{ secrets.GITHUB_TOKEN }}

      - name: Store benchmark results
        uses: actions/cache/save@v4
        # only store the cache if being run on main
        if: github.ref == 'refs/heads/main'
        with:
          path: ./cache
          key: ${{ matrix.os }}-benchmark