Uncompressing the cache is super slow

[MatthieuVegreville]

Hello,

Thank you for your work on actions/cache. We have a cache of almost 2GB. The download time is about 15s and uncompressing it takes around 1min45s.

Is there anyway to speed this uncompressing part? It seems very long for such a size.

Thanks, Matthieu

[Limpisey168]

3214fb2af73f94d4443d8cadebe3f51567595f02

[bishal-pdMSFT]

@MatthieuVegreville that's really slow de-compression.

1. Does this slowness happen for other large sized caches as well?
2. Does this slowness happen consistently?
3. Are you using github hosted runner or self-hosted one?

[aschillio]

Hello @bishal-pdMSFT,

I'm working with Matthieu on this topic.

1. It's only used in our monorepos, we have no other repo where we use the cache.
2. Yes, it does.
3. github hosted

Do you have any idea what could cause these ?

You would find below our cache conf. Perhaps the wildcard system is not working properly ?

      - uses: actions/checkout@v2
      - name: Get yarn cache directory path
        id: yarn-cache-dir-path
        run: |
          echo "::set-output name=dir::$(yarn cache dir)"
          echo "::set-output name=version::$(yarn -v)"
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v2
        with:
          node-version: '16.x'
      - uses: actions/cache@v2
        with:
          path: |
            **/node_modules
            **/.eslintcache
            ${{ steps.yarn-cache-dir-path.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

Thanks for taking the time.

[bishal-pdMSFT]

No, I don't have any idea right now. May be something to do with size or number of files in the package. is the repo public to take a quick look at yarn file?

@vsvipul you recently ran some numbers on the de-compression perf. Was a large package (~2GB) part of that test?

[vsvipul]

@bishal-pdMSFT No, but 130 files of 6.6 GB took took around 113 seconds to decompress. So, for 2 GB it should be comparatively lesser.

@aschillio Is the runner a mac runner? Because mac runners run relatively slower than windows / linux runner.

[aschillio]

@bishal-pdMSFT No repo is not public. I can share with you the package.json if it helps. Note we are using mono repo with yarn workspace.

@vsvipul Not, all our runners are linux based.

Let me know if you need any more info to investigate this. Thanks a lot guys.

Front:

{
  "name": "@offspend/web",
  "version": "3.1.0",
  "private": true,
  "browserslist": {
    "production": [
      ">0.2%",
      "ie 11",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "ie 11",
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "resolutions": {
    "webpack": "4.44.2"
  },
  "dependencies": {
    "@auth0/auth0-spa-js": "^1.11.0",
    "@date-io/core": "^2.8.0",
    "@date-io/moment": "^1.3.13",
    "@js-joda/core": "^3.2.0",
    "@material-ui/core": "^4.11.0",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.56",
    "@material-ui/pickers": "^3.2.10",
    "@material-ui/styles": "^4.10.0",
    "@mui/icons-material": "^5.0.5",
    "@promises/-all": "^0.5.0",
    "@sentry/react": "^5.22.3",
    "@sentry/tracing": "^5.22.3",
    "ag-grid-community": "^26.1.0",
    "ag-grid-react": "^26.1.0",
    "axios": "^0.19.2",
    "babel-plugin-relay": "^11.0.2",
    "browser-image-compression": "^1.0.15",
    "change-case": "^4.1.1",
    "chart.js": "^2.9.3",
    "clsx": "^1.1.1",
    "csv-stringify": "^5.5.1",
    "downloadjs": "^1.4.7",
    "firebase": "^7.17.1",
    "formik": "^2.1.5",
    "fuse.js": "^6.4.6",
    "history": "^4.10.1",
    "i18n-iso-countries": "^6.8.0",
    "i18next": "^19.7.0",
    "i18next-browser-languagedetector": "^6.0.1",
    "i18next-http-backend": "^1.0.18",
    "immer": "^6.0.9",
    "js-base64": "^3.5.2",
    "js-cookie": "^2.2.1",
    "jss": "^10.3.0",
    "jss-rtl": "^0.3.0",
    "jwt-decode": "^2.2.0",
    "lodash": "^4.17.19",
    "moment": "^2.28.0",
    "moment-with-locales-es6": "^1.0.1",
    "notistack": "^0.9.17",
    "nprogress": "^0.2.0",
    "prismjs": "^1.20.0",
    "prop-types": "^15.7.2",
    "react": "^16.13.1",
    "react-app-polyfill": "^1.0.6",
    "react-chartjs-2": "^2.9.0",
    "react-component-export-image": "^1.0.5",
    "react-dom": "^16.13.1",
    "react-draft-wysiwyg": "^1.14.5",
    "react-dropzone": "^10.2.2",
    "react-feather": "^2.0.8",
    "react-helmet": "^5.2.1",
    "react-highlight-words": "^0.17.0",
    "react-i18next": "^11.7.2",
    "react-modal-image": "^2.5.0",
    "react-number-format": "^4.7.3",
    "react-perfect-scrollbar": "^1.5.8",
    "react-relay": "^11.0.2",
    "react-router": "^5.2.0",
    "react-router-dom": "^5.2.0",
    "relay-runtime": "^11.0.2",
    "smartlook-client": "^5.0.0",
    "yup": "^0.28.5",
    "yup-password": "^0.2.1"
  },
  "devDependencies": {
    "@babel/core": "^7.14.3",
    "@babel/runtime": "^7.14.0",
    "@storybook/addon-actions": "^6.3.9",
    "@storybook/addon-essentials": "^6.3.9",
    "@storybook/addon-links": "^6.3.9",
    "@storybook/node-logger": "^6.3.9",
    "@storybook/preset-create-react-app": "^3.2.0",
    "@storybook/react": "^6.3.9",
    "@types/history": "^4.7.7",
    "@types/jest": "^25.2.3",
    "@types/js-cookie": "^2.2.6",
    "@types/jsonwebtoken": "^8.5.0",
    "@types/jwt-decode": "^2.2.1",
    "@types/lodash": "^4.14.158",
    "@types/node": "^15.0.2",
    "@types/nprogress": "^0.2.0",
    "@types/react": "^16.9.43",
    "@types/react-beautiful-dnd": "^12.1.3",
    "@types/react-dom": "^16.9.8",
    "@types/react-draft-wysiwyg": "^1.13.0",
    "@types/react-helmet": "^5.0.16",
    "@types/react-router-dom": "^5.1.5",
    "@types/uuid": "^7.0.4",
    "@types/yup": "^0.28.3",
    "@typescript-eslint/eslint-plugin": "^4.22.1",
    "@typescript-eslint/parser": "^4.22.1",
    "babel-loader": "8.1.0",
    "eslint": "^7.28.0",
    "eslint-config-airbnb": "^18.2.1",
    "eslint-config-airbnb-typescript": "^12.3.1",
    "eslint-config-prettier": "^8.3.0",
    "eslint-import-resolver-webpack": "^0.13.0",
    "eslint-plugin-import": "^2.22.1",
    "eslint-plugin-json-format": "^2.0.1",
    "eslint-plugin-jsx-a11y": "^6.3.1",
    "eslint-plugin-mdx": "^1.7.1",
    "eslint-plugin-prettier": "3.4.0",
    "eslint-plugin-react": "^7.20.3",
    "eslint-plugin-react-hooks": "^4.2.0",
    "graphql": "^15.5.0",
    "mdx-loader": "^3.0.2",
    "numeral": "^2.0.6",
    "prettier": "^2.2.1",
    "react-scripts": "^4.0.3",
    "relay-compiler": "^11.0.2",
    "relay-compiler-language-typescript": "^14.0.0",
    "relay-config": "^11.0.2",
    "typescript": "^4.2.4"
  },
  "bundledDependencies": [
    "greenly-shared"
  ],
  "engines": {
    "node": "16.x"
  },
  "licence": "UNLICENSED",
  "proxy": "http://localhost:3000"
}

Back:

{
  "name": "@offspend/api",
  "version": "0.1.0",
  "private": true,
  "description": "OffSpend Back-end",
  "workspaces": {
    "nohoist": [
      "**"
    ]
  },
  "dependencies": {
    "@js-joda/core": "^3.2.0",
    "@js-joda/locale_en": "^3.2.2",
    "@js-joda/locale_fr": "^3.2.2",
    "@promises/-all": "^0.5.0",
    "@sentry/integrations": "^5.15.0",
    "@sentry/node": "^5.11.0",
    "@types/adm-zip": "^0.4.33",
    "@types/google-spreadsheet": "^3.1.2",
    "@types/json2csv": "^5.0.1",
    "@types/moment-timezone": "^0.5.12",
    "@types/papaparse": "^5.2.6",
    "@types/uuid": "^8.3.1",
    "adm-zip": "^0.5.1",
    "apple-auth": "^1.0.5",
    "aws-sdk": "^2.828.0",
    "axios": "^0.19.2",
    "axios-retry": "^3.1.8",
    "bcryptjs": "^2.4.3",
    "body-parser": "^1.19.0",
    "concurrently": "^5.1.0",
    "crypto-random-string": "^3.3.0",
    "date-fns": "^2.4.1",
    "decod": "^6.1.6",
    "deep-object-diff": "^1.1.0",
    "dotenv": "^8.2.0",
    "dotenv-cli": "^3.1.0",
    "express": "^4.17.1",
    "express-graphql": "^0.12.0",
    "fbgraph": "^1.4.4",
    "google-auth-library": "^5.7.0",
    "google-spreadsheet": "^3.1.15",
    "googleapis": "^74.2.0",
    "graphql": "^15.5.0",
    "graphql-iso-date": "^3.6.1",
    "graphql-relay": "^0.6.0",
    "graphql-upload": "^12.0.0",
    "greenly-shared": "1.0.0",
    "iconv-lite": "^0.6.3",
    "isemail": "^3.2.0",
    "jschardet": "^3.0.0",
    "json2csv": "^5.0.6",
    "jsonwebtoken": "^8.5.1",
    "lodash": "^4.17.15",
    "moment": "^2.24.0",
    "moment-timezone": "^0.5.27",
    "mongodb": "^4.2.2",
    "papaparse": "^5.3.1",
    "pg": "^8.6.0",
    "reflect-metadata": "^0.1.13",
    "stripe": "8.119.0",
    "ts-node": "^9.1.1",
    "typeorm": "^0.2.32",
    "typescript": "^4.2.4",
    "uuid": "^8.3.2",
    "winston": "^3.2.1",
    "xlsx": "^0.17.4"
  },
  "devDependencies": {
    "@types/bcryptjs": "^2.4.2",
    "@types/express": "^4.17.11",
    "@types/graphql-iso-date": "^3.4.0",
    "@types/graphql-relay": "^0.6.0",
    "@types/graphql-upload": "^8.0.7",
    "@types/jest": "^24.0.21",
    "@types/jsonwebtoken": "^8.3.4",
    "@types/lodash": "^4.14.142",
    "@types/node": "^15.6.0",
    "@typescript-eslint/eslint-plugin": "^4.15.2",
    "@typescript-eslint/parser": "^4.15.2",
    "@typescript-eslint/typescript-estree": "^4.15.2",
    "eslint": "^7.28.0",
    "eslint-config-airbnb": "^18.0.1",
    "eslint-config-prettier": "^6.7.0",
    "eslint-plugin-import": "^2.23.4",
    "eslint-plugin-json-format": "^2.0.1",
    "eslint-plugin-prettier": "3.4.0",
    "jest": "^26.6.3",
    "nodemon": "^1.19.3",
    "ts-jest": "^26.5.6"
  },
  "engines": {
    "node": "16.x"
  }
}

[aschillio]

@vsvipul any news ?

[vsvipul]

@aschillio Unfortunately, looks like it has something to do with your repo and the data cached for your repo only. Maybe the number of files are very large. Can you try these-

• Creating a sample repo to generate files of similar size and try caching them and compare times with this repo.
• Compress the same files locally and decompress using zstd and compare the time with github hosted runner.

[aschillio]

@vsvipul So you don't see any error in our configuration ?

Ok I can try that, I would let you know. But I'm not sure we are the only one hitting the issue.

[vsvipul]

@aschillio config looks fine you can try putting in absolute paths instead of **/path .

[aschillio]

@vsvipul I would try that. Also, don't you find the download bandwith is pretty slow ? 80mb/s to 130mb/s. I already seen sometimes 200mb/s on other runner. Is there an explanation about that ?

[vsvipul]

@aschillio No, I normally get 120-140 MBps only. It can go upto 200 for smaller caches, but for large caches, average is 120-140 only.

[aschillio]

@vsvipul Ok thanks for the info.

Can you share the conf you used for the test caching you have done ?

[aschillio]

Ok so I tried using relative path, and the caching result are absolutely not the same.

With my previous conf, I had a cache size of 1700mb. Now that I added relative path for the 4 node_modules folder we have, the cache size went down to 459mb, and thus way faster. The install step seems fine with that.

Do you have any explanation ? I'm not sure I'm getting what is happening here.

[vsvipul]

The only explanation I can think of is that **/node_modules is searching recursively for directories with name node_modules inside all the directories. If it is given an absolute path, it doesn't need to todo that and should be fast.

I'm not sure why the size reduced? Maybe node_modules inside node_modules are being cached multiple times, which is causing this. Glad to see that these are much faster for your now.

@bishal-pdMSFT @ashwinsangem Can you also please take a look at this?

[aschillio]

Yes, you are probably right, the recursive pattern is maybe going every deps and getting each node modules of each dependencies. Which I can understand might be a lot to take for the runner.

[vsvipul]

Now that this is fixed, going ahead and closing this. Thank you for opening this. Feel free to reach out if you see any more issues.