c-eliasson commented 4 years ago

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: CacheV2 (https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/CacheV2)

Environment

Server - Azure Pipelines
- Account name: markethype
- Team project name: MarketHype
- Build ID: 1563
- URL: https://dev.azure.com/markethype/MarketHype/_build/results?buildId=1563
Agent - Hosted:
- Agent Queue name: Azure Pipelines (https://dev.azure.com/markethype/MarketHype/_settings/agentqueues?queueId=17&view=jobs)

Issue Description

Using the CacheV2 task to cache Yarn packages, we're getting an error in the post-job step of the cache task.

The task is defined exactly as in the docs.

variables:
  YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

steps:
- task: Cache@2
  inputs:
    key: 'yarn | "$(Agent.OS)" | yarn.lock'
    restoreKeys: |
       yarn | "$(Agent.OS)"
    path: $(YARN_CACHE_FOLDER)
  displayName: Cache Yarn packages

We're using the vs2017-win2016 image.

Error logs

Starting: Cache Yarn packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - yarn                         [string]
 - "Windows_NT"                 [string]
 - common/config/rush/yarn.lock [file] --> 0551FA989D7CDCB2101F1397E8E7C5468396D4C72935548B6A74BEB126F0296F
Resolved to: yarn|"Windows_NT"|57F7K6kT3NtH/ZuSuWsxueTn4wCxuYls973WEnqCJgE=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 06101a50-81b0-41b6-ab13-b130ee5c98ae
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `yarn|"Windows_NT"|57F7K6kT3NtH/ZuSuWsxueTn4wCxuYls973WEnqCJgE=`
Information, There is a cache miss.
tar: D\:\\a\\1\\.yarn: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 06101a50-81b0-41b6-ab13-b130ee5c98ae
##[error]Process returned non-zero exit code: 2
Finishing: Cache Yarn packages

jmaragon commented 4 years ago

Same issue here, but this time with npm

Starting: Cache npm Task : Cache Description : Cache files between runs Version : 2.0.1 Author : Microsoft Corporation Help : https://aka.ms/pipeline-caching-docs Resolving key:

npm [string]
"Windows_NT" [string]
./product/package.json [file] --> 66FF0092057590D58252E7A1365E9520D41F83083CE0C8C1DA0E038DC4C2D9D1 Resolved to: npm|"Windows_NT"|2DZ3WcfStD108j8qipq/0v97TF5eyy6fpvtPYfpDrZM= Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session xxxxxxxxxx-1282-485c-a6f4-b92a59ed4955 Information, Getting a pipeline cache artifact with one of the following fingerprints: Information, Fingerprint: npm|"Windows_NT"|2DZ3WcfStD108j8qipq/0v97TF5eyy6fpvtPYfpDrZM= Information, There is a cache miss. tar: Cannot change to directory E:\Agents\3\283\s\node_modules: No such file or directory tar: Error is not recoverable: exiting now Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session XXXXXXXX282-485c-a6f4-b92a59ed4955
[error]Process returned non-zero exit code: 2

Finishing: Cache npm

bradringel commented 4 years ago

Same issue here on a hosted linux agent


2020-06-09T21:30:32.6150811Z ##[section]Starting: Cache yarn packages
2020-06-09T21:30:32.6163777Z ==============================================================================
2020-06-09T21:30:32.6164193Z Task         : Cache
2020-06-09T21:30:32.6164523Z Description  : Cache files between runs
2020-06-09T21:30:32.6164828Z Version      : 2.0.1
2020-06-09T21:30:32.6165145Z Author       : Microsoft Corporation
2020-06-09T21:30:32.6165519Z Help         : https://aka.ms/pipeline-caching-docs
2020-06-09T21:30:32.6165941Z ==============================================================================
2020-06-09T21:30:32.9975064Z Resolving key:
2020-06-09T21:30:33.0154988Z  - yarn      [string]
2020-06-09T21:30:33.0162685Z  - "Linux"   [string]
2020-06-09T21:30:33.0743802Z  - yarn.lock [file] --> EA85735E476B7CE9FF55138A52CE89B7299F92856B621075773F035440C87D93
2020-06-09T21:30:33.0785039Z Resolved to: yarn|"Linux"|QlvZU5gmusF+bBuImFOlW68Qf4aNmdJ1pmhH2LChb5o=
2020-06-09T21:30:33.6622757Z Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session c6edd63d-1238-44dd-a471-3439ac2f402e
2020-06-09T21:30:33.6861181Z Information, Getting a pipeline cache artifact with one of the following fingerprints:
2020-06-09T21:30:33.6862267Z Information, Fingerprint: `yarn|"Linux"|QlvZU5gmusF+bBuImFOlW68Qf4aNmdJ1pmhH2LChb5o=`
2020-06-09T21:30:34.2316763Z Information, There is a cache miss.
2020-06-09T21:30:34.2834153Z tar: /home/vsts/work/1/.yarn: Cannot open: No such file or directory
2020-06-09T21:30:34.2834821Z tar: Error is not recoverable: exiting now
2020-06-09T21:30:34.5635719Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session c6edd63d-1238-44dd-a471-3439ac2f402e
2020-06-09T21:30:34.5689658Z ##[error]Process returned non-zero exit code: 2
2020-06-09T21:30:34.5844564Z ##[section]Finishing: Cache yarn packages```

stevewarsa commented 4 years ago

Same issue here with yarn - it is in some autogenerated task after my WAR deploy is done - the task is called "Post-job: Cache":

tar: /blah/blahgent/_work/1/s//node_modules, !/node_modules/**/node_modules: Cannot open: No such file or directory tar: Error is not recoverable: exiting now Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 18de2057-e96c-4ff9-9091-595b9595f5cf

[error]Process returned non-zero exit code: 2

Finishing: Cache

Any clue how to solve?

lesterchan commented 4 years ago

Came in here to +1 to this, below is my config:

variables:
  - name: yarnCacheFolder
    value: $(Pipeline.Workspace)/.yarn

  - task: Cache@2
    displayName: Cache Yarn Packages
    inputs:
      key: yarn | $(Agent.OS) | yarn.lock
      restoreKeys: |
        yarn | $(Agent.OS)
        yarn
      path: $(yarnCacheFolder)

Post-job: Cache Yarn Packages will throw an error

/home/vsts/work/1/.yarn: Cannot open: No such file or directory

jfheins commented 4 years ago

For me it works, but I have set the directory directly because I found no suitable pipeline variable:

      - task: Cache@2
        inputs:
          key: 'yarn | "$(Agent.OS)" | Frontend/.../yarn.lock'
          restoreKeys: yarn | "$(Agent.OS)"
          path: /home/vsts/.cache/yarn
        displayName: Cache Yarn packages

I'm using a hosted agent (ubuntu-18.04); maybe this works for you as well. One can print the relevant directory for yarn with yarn cache dir

jotatoledo commented 4 years ago

@jfheins thanks for the hint.

Current configuration works for me in cloud-hosting:

pool:
  vmImage: 'ubuntu-latest'
variables:
 - name: YARN_CONFIG_CACHE
    value: /home/vsts/.cache/yarn/v6
...
- task: Cache@2
  displayName: Cache node modules
  inputs:
    key: 'yarn | "$(Agent.OS)" | yarn.lock'
    restoreKeys: |
      yarn | "$(Agent.OS)"
    path: $(YARN_CONFIG_CACHE)

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

bradringel commented 3 years ago

Bumping because this is still happening and I don't think it should be closed

sheldonlynn commented 3 years ago

I was running into this exact issue and fixed it by adding YARN_CACHE_FOLDER to the pipeline variables.

I had the Cache task as part of a task group and in the pipeline that uses the task group, there is a required field that asks for the value of YARN_CACHE_FOLDER which is used as 'path' for the Cache task. Filled in the field with $(Pipeline.Workspace)/.yarn but the post-job kept failing with same error as above.

Once I also defined YARN_CACHE_FOLDER=$(Pipeline.Workspace)/.yarn in the pipeline variables, the post-job stopped failing.

rfink commented 3 years ago

Bump - this actually just started happening on a build that hasn't changed since December.

loxy commented 3 years ago

Same here for Maven:


variables:
  IS_MASTER_OR_DEVELOP: $[in(variables['Build.SourceBranch'], 'refs/heads/master', 'refs/heads/develop')]
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: Cache@2
    displayName: "Cache Maven local repo"
    inputs:
      key: 'maven | "$(Agent.OS)" | **/pom.xml'
      restoreKeys: |
        maven | "$(Agent.OS)"
        maven
      path: $(MAVEN_CACHE_FOLDER)

  - task: Maven@3
    displayName: "Publish artifacts"
    condition: eq(variables.IS_MASTER_OR_DEVELOP, true)
    inputs:
      mavenPomFile: './pom.xml'
      mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
      mavenAuthenticateFeed: true
      goals: 'deploy'

kevinpiske commented 2 years ago

Same as @loxy

sondretj commented 2 years ago

I'm not sure why the pipeline I created started throwing the /home/vsts/work/1/.yarn: Cannot open: No such file or directory error as I use the exact same setup other places, with $(Pipeline.Workspace)/.yarn. I tried using different vmImages with no luck. Here's how I solved it for the ones interested.

          - bash: echo "##vso[task.setvariable variable=yarnCacheDir;]$(yarn cache dir)"
            displayName: Set yarn cache dir variable
          - task: Cache@2
            inputs:
              key: '"yarn" | "$(Agent.OS)" | yarn.lock'
              restoreKeys: |
                yarn | "$(Agent.OS)"
              path: $(yarnCacheDir)
            displayName: Cache Yarn packages

pradpant commented 2 years ago

Same here for Maven:

variables:
  IS_MASTER_OR_DEVELOP: $[in(variables['Build.SourceBranch'], 'refs/heads/master', 'refs/heads/develop')]
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: Cache@2
    displayName: "Cache Maven local repo"
    inputs:
      key: 'maven | "$(Agent.OS)" | **/pom.xml'
      restoreKeys: |
        maven | "$(Agent.OS)"
        maven
      path: $(MAVEN_CACHE_FOLDER)

  - task: Maven@3
    displayName: "Publish artifacts"
    condition: eq(variables.IS_MASTER_OR_DEVELOP, true)
    inputs:
      mavenPomFile: './pom.xml'
      mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
      mavenAuthenticateFeed: true
      goals: 'deploy'

@loxy Any Suggestion ? I'm also stuck here .

loxy commented 2 years ago

@pradpant No, didn't found a solution

adriansergheev commented 2 years ago

bump!

daerogami commented 2 years ago

This was working in a single stage pipeline, once I added another stage that causes this to get called twice, I now get the error with the cache post-job Local agent running Win10 pro

Starting: Cache Yarn packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.1
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - yarn                          [string]
 - "Windows_NT"                  [string]
 - src/MyProject.Web.Host/yarn.lock [file] --> A51A47DAA3D812F7DBE1028017AC2CA60169EA036A0C8BDD2DE0B6DADB8B21FE
Resolved to: yarn|"Windows_NT"|+Z30KjYR/75Haaf6MF5eU1mX/TrlZ7I5cG6FuT0QJfE=
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 7d701a80-cde5-40c0-abed-7f66814cc117
Getting a pipeline cache artifact with one of the following fingerprints:
Fingerprint: `yarn|"Windows_NT"|+Z30KjYR/75Haaf6MF5eU1mX/TrlZ7I5cG6FuT0QJfE=`
There is a cache miss.
tar: could not chdir to 'C:\agent\_work\5\.yarn'

ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 7d701a80-cde5-40c0-abed-7f66814cc117
##[error]Process returned non-zero exit code: 1
Finishing: Cache Yarn packages

nadworny commented 2 years ago

Same here. Seems the cache plugin is broken. Found another solution here: https://xebia.com/blog/caching-your-node-modules-in-azure-devops/

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

loxy commented 2 years ago

/remove-stale

TiceWise commented 1 year ago

I'm also running into this issue, also with two consecutive @Cache2 tasks (one for $(Pipeline.Workspace)/.npm and one for $(Pipeline.Workspace)/.cache/Cypress). Also tried with CacheBeta@1, same issue. The first post-job executed is Cache Cypress (not npm which is the first cache task), which fails with:

There is a cache miss.
tar: /agent/_work/1/.cache/Cypress: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session ***-**-**-**-***
##[error]Process returned non-zero exit code: 2
Finishing: Cache Cypress binary

Bengkel commented 1 year ago

I think caching will only work on a self hosted agent?

jfheins commented 1 year ago

I think caching will only work on a self hosted agent?

No, this task is especially good for the Azure hosted agents. In a self hosted scenario, most of the caching works without this task. (For example: yarn has a machine wide global cache that persists in a self hosted agent)

The tricky thing is to get the cache directory right. For example, an Azure agent mighthave the yarn cache in "/home/vsts/.cache/yarn" while a self hosted might need "/home/AzDevOps/.cache/yarn"

I'd recommend to include a step that logs out the cache directories for easier troubleshooting:

      - script: |
          node -v
          npm -v
          yarn -v
          yarn cache dir
          npm config get cache
        displayName: "Log versions & paths"

esoxjem commented 1 year ago

Solution

So, the issue seems to be that $(Pipeline.Workspace) does not return the home directory and hence the step can't find the build folders that are typically in the home dir. Take the output from the command @jfheins posted and add that to the variable directly.

# example for .npm

# replace this
variables:
  CACHE_FOLDER: $(Pipeline.Workspace)/.npm

# with this
variables:
  CACHE_FOLDER: /home/vsts/.npm

Result

Here's the result of the successful run after making the change

Screenshot 2023-02-08 at 21 21 36

vixero-dev commented 1 year ago

I have the same error, but with pnpm. This is the yaml from the docs:

variables:
  pnpm_config_cache: $(Pipeline.Workspace)/.pnpm-store

steps:
  - task: Cache@2
    inputs:
      key: 'pnpm | "$(Agent.OS)" | pnpm-lock.yaml'
      path: $(pnpm_config_cache)
    displayName: Cache pnpm

  - script: |
      corepack enable
      corepack prepare pnpm@latest-8 --activate
      pnpm config set store-dir $(pnpm_config_cache)
    displayName: "Setup pnpm"

  - script: |
      pnpm install
      pnpm run build
    displayName: "pnpm install and build"

johnterickson commented 1 year ago

@vixero-dev Can you share your the log from the task failure?

A lot of these issues look like folks either aren't:

Setting the correct variable (as defined by the package manager) at the pipeline-level to configure where the package manager should cache things
Actually invoking the package manager (the cache task does not do this)

vixero-dev commented 1 year ago

@johnterickson you are correct. The variable wasn't being set.

After changing this part:

  - script: |
      corepack enable
      corepack prepare pnpm@latest-8 --activate
      pnpm config set store-dir $(pnpm_config_cache)
    displayName: "Setup pnpm"

to this:

  - script: |
      corepack enable
      corepack prepare pnpm@latest-8 --activate
    displayName: "setup pnpm"

  - script: pnpm config set store-dir $(pnpm_config_cache)
    displayName: "setup pnpm config"

I stopped experiencing this issue.

dombarnes commented 10 months ago

I had some similar issues that ended up being due to agents getting different versions of yarn. I added

- bash: |
        corepack enable
        echo "Requesting Yarn version $(YARN_VERSION)"
        yarn set version $(YARN_VERSION)
        echo "Running Yarn $(yarn --version)"

along with configuring the YARN_VERSION variable to 3.6.4 in my case. I found out some agents would give me 1.22 sometimes in some build steps and 3.6 in others. This helped me diagnose this.

yooakim commented 9 months ago

Also experience this issue when following Microsoft's documentation. It is an issue with the paths. This blog posts is a great dive into how to fix this and the Cache task.

nickshanks347 commented 7 months ago

Try adding an s between $(Pipeline.Workspace) and .yarn, so variable goes from:

YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

To:

YARN_CACHE_FOLDER: $(Pipeline.Workspace)/s/.yarn

This is because when using checkout: self, it checks out the repository to $(Pipeline.Workspace)/s, as is explained here. Unless your using magic, your .yarn folder should be inside the repository.

DxTa commented 2 months ago

@nickshanks347 's comment is really helpful. In deed, the same also happening to other tool such as pip

Evanion commented 2 weeks ago

I tried

variables:
  NPM_CACHE_FOLDER: $(Pipeline.Workspace)/s/.npm
# ...
steps:
  - task: Cache@2
     displayName: Cache NPM packages
     inputs:
       key: '"npm" | "$(Agent.OS)" | package-lock.json'
       restoreKeys: |
         "npm" | "$(Agent.OS)"
         "npm"
       path: $(NPM_CACHE_FOLDER)
       cacheHitVar: NPM_CACHE_RESTORED

But I get

Post-job: Cache NPM packages

[View raw log](https://dev.azure.com/landshypotek/980d789b-7245-4c89-a4a0-320898842fa4/_apis/build/builds/9637/logs/19)
Starting: Cache NPM packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.198.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - "npm"             [string]
 - "Linux"           [string]
 - package-lock.json [file] --> 0C9C69F14CFB03BABFC87109EECCB2B787FBAD0ECFC7E1F9F823A953E0A5303E
Resolved to: "npm"|"Linux"|NyzAW704Sn4J2Y1PCdbgFC5xaendrqBTYW+d+xSeAhY=
Using default max parallelism.
Using default max parallelism.
Max dedup parallelism: 192
DomainId: 0
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session ab82fec9-6c06-4dc7-9d91-1f0827ba36b5
Hashtype: Dedup64K
Getting a pipeline cache artifact with one of the following fingerprints:
Fingerprint: `"npm"|"Linux"|NyzAW704Sn4J2Y1PCdbgFC5xaendrqBTYW+d+xSeAhY=`
There is a cache miss.
tar: /home/vsts/work/1/s/.npm: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session ab82fec9-6c06-4dc7-9d91-1f0827ba36b5
##[error]Process returned non-zero exit code: 2
Finishing: Cache NPM packages

I would expect the post-job to create the folder if it doesn't exist.

jfheins commented 2 weeks ago

I would expect the post-job to create the folder if it doesn't exist.

The output is telling you that the directory with the stuff you want to cache does not exist. (i.e. there is nothing to cache) Have you tried to run npm --global cache verify in the pipeline to have npm output the cache directory?

microsoft / azure-pipelines-tasks

Error in post-job step of CacheV2 task when caching yarn packages #12892

Required Information

Environment

Issue Description

Error logs

Same issue here, but this time with npm

[error]Process returned non-zero exit code: 2

Same issue here on a hosted linux agent

[error]Process returned non-zero exit code: 2

Solution

Result