Cache hits don't occur in Github Actions workflows

[sammccord]

What version of Turborepo are you using?

1.0.23

What package manager are you using / does the bug impact?

npm / pnpm

What operating system are you using?

Linux

Describe the Bug

When attempting to achieve full turbo speeds in GHA workflows by caching turbo's cache with actions/cache@v2, the restored cache isn't respected in subsequent workflow runs, and results in misses even when there aren't any changes to source files.

This occurs regardless of the --cache-dir arg, or if packages' .turbo/*.logs are included in the cache.

<3 this project btw

Expected Behavior

Turbo should use the cache folder restored by actions/cache@v2

To Reproduce

Here's a basic reproduction using the default template - https://github.com/sammccord/ga-repro

If you check out this run, when I build with a cache that should be valid, it instead misses all packages and creates new entries in the cache.

I can't reproduce on my Linux box or Mac lappy, everything works peachy there.

[michael-land]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          cache: 'yarn'
          cache-dependency-path: yarn.lock
          node-version: '16'

      - name: Turbo Cache
        id: turbo-cache
        uses: actions/cache@v2
        with:
          path: .turbo
          key: turbo-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
          restore-keys: |
            turbo-${{ github.job }}-${{ github.ref_name }}-

      - name: Install dependencies
        run: |
          npm install -g yarn
          yarn
      - name: Build
        run: |
          yarn run build --cache-dir=".turbo"

[sammccord]

@xiaoyu-tamu What you're suggesting is a workaround, but it's still a bug that turbo's caching does not work consistently between package managers the docs claim to support.

[kieranm]

@xiaoyu-tamu This workaround worked for me, but only if I specified...

with:
  path: node_modules/.cache/turbo

instead of:

with:
  path: .turbo

[nicksrandall]

Also seeing this.

[luismramirezr]

I solved it by defining the cache-dir outside the node_modules directory. If I let the cache dir inside the node_modules, the hash changes between every turbo run.

   - name: Turbo Cache
        id: turbo-cache
        uses: actions/cache@v2
        with:
          path: .turbo
          key: turbo-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
          restore-keys: |
            turbo-${{ github.job }}-${{ github.ref_name }}-

and then when running the turbo commands: yarn turbo run <pipeline> --cache-dir=.turbo

[tknickman]

@sammccord are you still running into this issue?

We also have docs now with a recipe for github actions using the remote cache.

[teebszet]

@tknickman are there any docs for this without using the remote cache?

[wh1t3cAt1k]

@tknickman I am still seeing this. Definitely interested in solving this using the local cache and not the remote one.

I'm afraid the workaround mentioned by @xiaoyu-tamu is not appropriate for me because I have root-level tasks whose inputs depend on .ts files.

Unfortunately if I set up the solution by @xiaoyu-tamu the cache never hits because .turbo directory just keeps getting new .ts files as cached artifacts.

--ignore flag did not solve the issue for me.

Therefore I am now stuck between two non-working solutions both of which result in constant cache misses :)

[michael-land]

The workaround that I posted was 7 months ago, turborepo now have a dedicate page for CI integration.

[wh1t3cAt1k]

Although I might as well be an idiot, because I just discovered that I ran actions/cache restoration prior to running npm ci which wipes out the node_modules folder.

I am checking this hypothesis right now.

[wh1t3cAt1k]

@xiaoyu-tamu, yes, I am aware and have diligently read through the documentation - unfortunately there is nothing there that says how to use native GitHub Actions caching as opposed to Vercel's caching.

Not sure if there might be some conflict of interest since Vercel offers remote caching as their paid product...

[wh1t3cAt1k]

For the record: I was able to fix the issue on my end using actions/cache.

The root cause, indeed, was because I first restored the cache and then did an npm ci, which wiped out the whole node_modules folder including the cache.

The solution is to either use npm i instead, or to simply reorder the actions: make sure that cache action comes after npm ci.

[attila]

I have configured all our turbo commands to use a dedicated cache folder outside node_modules, e.g. --cache-dir=./.turbo-cache. I wish it was configurable via environment variables but still easy to work around the fact that "npm ci" removes node_modules