[Feature] Does Yarn 2 have any sort of global cache?

[kirill-konshin]

Describe the user story

Yarn 1 had a useful feature — it stored global cache, for example in folder /usr/local/share/.cache/yarn, which could be revealed by running yarn cache dir.

Having cache like this allowed to mount it to CI runner so that all jobs on runner will have shared cache, and it will be much faster rather than to cache Yarn's cache per-job.

Yarn 2 does not have such command.

Describe the solution you'd like

Same thing as before. Global cache which will be used as fallback if project's .yarn/cache is empty.

[arcanis]

• enableGlobalCache
• cacheFolder

Note that if you're under OSX enabling the global cache isn't very useful due to the use of copy-on-write (all downloaded files are stored in the global cache first, and we clone them into the local cache afterwards; in result all per-project caches from a single computer user effectively contain a single copy of each package).

[kirill-konshin]

But how to determine where will the global cache be located?

Pipelines run on Linux, just in case.

[arcanis]

The default path is system-specific, but you can set it with globalFolder.

[kirill-konshin]

Got the an error with Storybook 6.0.0-alpha.14 and following settings:

$ yarn config set enableGlobalCache true
$ yarn config set globalFolder /usr/local/share/.cache/yarn/yarn2

storybook: info => Loading presets
storybook: WARN   Failed to load preset: "/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-preset.js"
storybook: ERR! Error: Cannot find module '@storybook/addons/package.json'
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR!     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
storybook: ERR!     at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)
storybook: ERR!     at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)
storybook: ERR!     at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)
storybook: ERR!     at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)
storybook: ERR!     at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)
storybook: ERR!     at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR!     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR!     at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR!     at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)
storybook: ERR!     at Module.require (internal/modules/cjs/loader.js:1040:19)
storybook: ERR!     at require (internal/modules/cjs/helpers.js:72:18)
storybook: ERR!     at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)
storybook: ERR!     at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR!     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR!     at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR!  Error: Cannot find module '@storybook/addons/package.json'
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR!     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
storybook: ERR!     at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)
storybook: ERR!     at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)
storybook: ERR!     at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)
storybook: ERR!     at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)
storybook: ERR!     at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)
storybook: ERR!     at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR!     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR!     at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR!     at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)
storybook: ERR!     at Module.require (internal/modules/cjs/loader.js:1040:19)
storybook: ERR!     at require (internal/modules/cjs/helpers.js:72:18)
storybook: ERR!     at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)
storybook: ERR!     at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR!     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR!     at Module.load (internal/modules/cjs/loader.js:1000:32) {
storybook: ERR!   stack: "Error: Cannot find module '@storybook/addons/package.json'\n" +
storybook: ERR!     'Require stack:\n' +
storybook: ERR!     '- /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js\n' +
storybook: ERR!     'Require stack:\n' +
storybook: ERR!     '- /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js\n' +
storybook: ERR!     '    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)\n' +
storybook: ERR!     '    at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)\n' +
storybook: ERR!     '    at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)\n' +
storybook: ERR!     '    at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)\n' +
storybook: ERR!     '    at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)\n' +
storybook: ERR!     '    at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)\n' +
storybook: ERR!     '    at Module._compile (internal/modules/cjs/loader.js:1151:30)\n' +
storybook: ERR!     '    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)\n' +
storybook: ERR!     '    at Module.load (internal/modules/cjs/loader.js:1000:32)\n' +
storybook: ERR!     '    at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)\n' +
storybook: ERR!     '    at Module.require (internal/modules/cjs/loader.js:1040:19)\n' +
storybook: ERR!     '    at require (internal/modules/cjs/helpers.js:72:18)\n' +
storybook: ERR!     '    at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)\n' +
storybook: ERR!     '    at Module._compile (internal/modules/cjs/loader.js:1151:30)\n' +
storybook: ERR!     '    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)\n' +
storybook: ERR!     '    at Module.load (internal/modules/cjs/loader.js:1000:32)',
storybook: ERR!   code: 'MODULE_NOT_FOUND',
storybook: ERR!   requireStack: [
storybook: ERR!     '/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js'
storybook: ERR!   ]
storybook: ERR! }

It works fine with local cache.

[kirill-konshin]

https://github.com/yarnpkg/berry/issues/965 something similar

[Js-Brecht]

Thanks @kirill-konshin. Makes more sense to leave my comment there

[kirill-konshin]

@arcanis looks like enableGlobalCache works weird for now.

By observing how Yarn 2 works, I see that it does cache dependencies somewhere in the system even when the option enableGlobalCache is not set. Is there a cross-platform way to determine where? I still would like to mount this directory from CI runner, but I'd like to keep enableGlobalCache setting per project...

[gabemeola]

enableGlobalCache seems misleading. Global caching is always enabled. Is this correct?

[arcanis]

Yes and no - without enableGlobalCache, the global cache will just be an indirection and the files will be copied into a per-project cache (depending on the system it can be zero-cost, thanks to copy-on-write).

With enableGlobalCache we directly reference the global cache into the generated PnP map, and don't generate the local cache. It presents some interesting quirks though, hence why it's not the default.

[kirill-konshin]

Hmmm... but this brings up my original question ))) I'd like to keep enableGlobalCache off (in order to keep PnP map to have only local) but I still would like to mount this indirect, system-dependent cache from container to the runner, so that all jobs will share it.

So is there a way to figure out it's location?

[phuctm97]

@kirill-konshin I share the same use case with you that I want to be able to save/attach cache to optimize CI/CD resources.

I ended up caching Yarn's local cache folder and everything works just fine. Why don't you do that, is there any complication or unseen issue, or you just didn't think of it?

Following is how I cache Yarn's local cache folder and the results:

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      node-version: 12.x
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Node.js ${{ env.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ env.node-version }}
      - id: find-yarn-cache-folder
        name: Find Yarn's cache folder
        run: echo "::set-output name=path::$(yarn config get cacheFolder)"
      - name: Cache Yarn's cache folder
        uses: actions/cache@v1
        with:
          path: ${{ steps.find-yarn-cache-folder.outputs.path }}
          key: yarn-cache-folder-os-${{ runner.os }}-node-${{ env.node-version }}-${{ hashFiles('yarn.lock') }}
          restore-keys: |
            yarn-cache-folder-os-${{ runner.os }}-node-${{ env.node-version }}-
            yarn-cache-folder-os-${{ runner.os }}-
      - name: Install dependencies
        run: yarn install
        ...

• Before applying cache:

  441 packages were already cached, 1556 had to be fetched

• After applying cache:

  1994 packages were already cached

Updated CI/CD still bundles my application properly and saves ~85% build time.

[phuctm97]

Btw, this issue seems not resolved yet, why is it closed, guys?

[kirill-konshin]

@phuctm97 in my case yarn config get cacheFolder returns me %PROJECT_DIR%/.yarn/cache which is project's folder, not system-wide cache folder.

@arcanis I think this issue has to be reopened, since it will give a lot of benefits for CI like Gitlab. There's still no solution how to determine which system-wide folder Yarn 2 is using. enableGlobalCache is a bit different, I need both:

1. local cache (to cache it using Gitlab, per branch)
2. global cache (same as in Yarn 1) to speed up when local cache is cold (first checkout of branch).

[arcanis]

Again, the global cache is in globalFolder.

[kirill-konshin]

Oh, so it's used even when enableGlobalCache is false? Sorry for confusion.

[qwIvan]

Is enableGlobalCache should disable on Linux with btrfs, but enable with ext4?