jenkinsci / jobcacher-plugin

Jenkins plugin that improves build performance for transient agents by caching files
https://plugins.jenkins.io/jobcacher/
MIT License
31 stars 32 forks source link

Cannot create cache as the path does not exist #120

Closed CSchulz closed 7 months ago

CSchulz commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.346.4.1 OS: Linux - 3.10.0-1160.76.1.el7.x86_64 --- Office-365-Connector:4.17.0 ace-editor:1.1 active-directory:2.25.1 allure-jenkins-plugin:2.30.3 analysis-model-api:10.13.0 android-emulator:3.1.3 android-signing:2.2.5 ansicolor:1.0.2 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 appcenter:0.11.1 async-http-client:1.7.24.3 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-cloudformation:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-codebuild:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ec2:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecr:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-elasticbeanstalk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-iam:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-logs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-minimal:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sns:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sqs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ssm:1.12.246-349.v96b_b_f7eb_a_c3c badge:1.9.1 basic-branch-build-strategies:1.3.2 blueocean:1.25.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-config:1.25.5 blueocean-core-js:1.25.5 blueocean-dashboard:1.25.5 blueocean-display-url:2.4.1 blueocean-events:1.25.5 blueocean-git-pipeline:1.25.5 blueocean-github-pipeline:1.25.5 blueocean-i18n:1.25.5 blueocean-jira:1.25.5 blueocean-jwt:1.25.5 blueocean-personalization:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-editor:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-5 bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-monitor-plugin:1.13+build.202205140447 build-time-blame:64.vd8f4018a2bbe build-timeout:1.21 build-view-column:0.3 buildtriggerbadge:251.vdf6ef853f3f5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-aborted-builds:1.19 cloudbees-administrative-monitors:1.0.4 cloudbees-analytics:1.42 cloudbees-assurance:2.276.0.23 cloudbees-aws-cli:1.5.20 cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_ cloudbees-blueocean-default-theme:0.8 cloudbees-console-log-masker:1.0.5 cloudbees-consolidated-build-view:1.6.1 cloudbees-even-scheduler:3.13 cloudbees-folder:6.729.v2b_9d1a_74d673 cloudbees-folders-plus:3.28 cloudbees-groovy-view:1.14 cloudbees-ha:4.39 cloudbees-jenkins-advisor:3.3.2 cloudbees-jsync-archiver:5.23 cloudbees-label-throttling-plugin:3.9 cloudbees-license:9.68 cloudbees-monitoring:2.14 cloudbees-nodes-plus:1.23 cloudbees-pipeline-policies:1.7 cloudbees-platform-common:1.17 cloudbees-plugin-usage:2.15 cloudbees-quiet-start:1.7 cloudbees-request-filter:1.7 cloudbees-ssh-slaves:2.18 cloudbees-support:3.29 cloudbees-template:4.56 cloudbees-uc-data-api:4.50 cloudbees-unified-ui:1.21 cloudbees-view-creation-filter:1.9 cloudbees-workflow-template:3.18 cloudbees-workflow-ui:2.8 cobertura:1.17 cocoapods-integration:0.2.0 code-coverage-api:3.0.5 command-launcher:84.v4a_97f2027398 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 conditional-buildstep:1.4.2 config-file-provider:3.10.0 configurationslicing:430.v966357576543 credentials:1129.vef26f5df883c credentials-binding:523.vd859a_4b_122e6 cucumber-reports:5.7.3 dark-theme:262.v0202a_4c8fb_6a data-tables-api:1.12.1-2 declarative-pipeline-migration-assistant:1.5.2 declarative-pipeline-migration-assistant-api:1.5.2 delivery-pipeline-plugin:1.4.2 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.10 docker-slaves:1.0.7 docker-workflow:1.29 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 email-ext:2.89 embeddable-build-status:237.v37e2c03964c9 envinject:2.881.v37c62073ff97 envinject-api:1.199.v3ce31253ed13 external-monitor-job:192.ve979ca_8b_3ccd favorite:2.4.1 font-awesome-api:6.1.1-1 forensics-api:1.15.1 git:4.11.5 git-changelog:3.25 git-client:3.11.1 git-parameter:0.9.18 git-server:1.11 git-tag-message:1.7.1 github:1.34.5 github-api:1.303-400.v35c2d8258028 github-branch-source:1656.v77eddb_b_e95df github-organization-folder:1.6 golang:1.4 google-oauth-plugin:1.0.6 google-play-android-publisher:4.2 gradle:1.39.4 greenballs:1.15.1 groovy-postbuild:2.5 h2-api:1.4.199 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hashicorp-vault-pipeline:1.4 hashicorp-vault-plugin:336.v182c0fbaaeb7 hidden-parameter:0.0.5 htmlpublisher:1.30 http_request:1.16 ignore-committer-strategy:1.0.4 infradna-backup:3.38.52 ionicons-api:31.v4757b_6987003 jackson2-api:2.13.3-285.vc03c0256d517 jacoco:3.3.2 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-4 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.5 jenkins-design-language:1.25.5 jira:3.7.1 jira-steps:2.0.141.vd0c6e6dc83f0 jjwt-api:0.11.5-77.v646c772fddb_0 jobConfigHistory:1176.v1b_4290db_41a_5 jobcacher:291.v1b_2284da_5dd7 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 junit-attachments:101.v82f494a_00e9e kpp-management-plugin:1.0.0 ldap:2.10 lockable-resources:2.18 mailer:414.vcc4c33714601 mapdb-api:1.0.9.0 mask-passwords:3.3 material-theme:0.5.2-rc100.6121925fe229 matrix-auth:3.1.2 matrix-project:772.v494f19991984 maven-plugin:3.19 mercurial:2.16.2 metrics:4.1.6.2 momentjs:1.1.1 monitoring:1.91.0 msbuild:1.30 nectar-license:8.41 nectar-rbac:5.75 next-build-number:1.8 nexus-jenkins-plugin:3.16.459.vcdf273b_29f8c node-iterator-api:1.5.1 nodejs:1.5.1 nodelabelparameter:1.11.0 oauth-credentials:0.5 okhttp-api:4.9.2-20211102 operations-center-agent:2.346.0.2 operations-center-analytics-config:2.222.0.1 operations-center-analytics-reporter:2.222.0.1 operations-center-client:2.346.0.2 operations-center-cloud:2.346.0.2 operations-center-context:2.346.0.8 pam-auth:1.8 parameterized-trigger:2.45 percentage-du-node-column:0.1.0 pipeline-aggregator-view:1.11 pipeline-aws:1.43 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-maven:1161.v89a_7dcec5d31 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-multibranch-defaults:2.1 pipeline-npm:0.9.2 pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.24 pipeline-timeline:1.0.3 pipeline-utility-steps:2.14.0 plain-credentials:1.8 plugin-usage-plugin:3.0 plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.5-2 port-allocator:1.8 prism-api:1.28.0-2 pubsub-light:1.16 rebuild:1.34 release:2.15 requests:2.2.19 resource-disposer:0.20 run-condition:1.5 sauce-ondemand:1.208 scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 scriptler:3.5 snakeyaml-api:1.30.2-76.vc104f7ce9870 sonar:2.14 splunk-devops:1.10.0 splunk-devops-extend:1.10.0 sse-gateway:1.25 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:291.v8211e4f8efb_c ssh-slaves:1.821.vd834f8a_c390e sshd:3.237.v883d165a_c1d3 structs:318.va_f3ccb_729b_71 support-core:1201.v8d1f54a_6ec7c terraform:1.0.10 text-file-operations:1.3.2 tfs:5.157.1 theme-manager:1.5 throttle-concurrents:2.10 timestamper:1.18 token-macro:293.v283932a_0a_b_49 trilead-api:1.67.vc3938a_35172f unique-id:2.2.1 user-activity-monitoring:1.7 variant:1.4 versioncolumn:2.2 warnings-ng:9.14.0 wikitext:3.15 windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1188.v0016b_4f29881 workflow-basic-steps:969.vc4ec3e4854b_f workflow-cps:2746.v0da_83a_332669 workflow-cps-checkpoint:2.13 workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1190.vc93d7d457042 workflow-job:1207.ve6191ff089f8 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:827.v7ef666c4d65c ws-cleanup:0.43 xray-connector:2.6.1 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux - 3.10.0-1160.76.1.el7.x86_64

Reproduction steps

cache(caches: [[$class: 'ArbitraryFileCache', cacheValidityDecidingFile: 'node_modules/@angular-devkit/build-angular/package.json', compressionMethod: 'ZIP', excludes: '', includes: '**/*', path: '.angular']], defaultBranch: '', maxCacheSize: 0) {
    // some block
}

Expected Results

Cache should be created without any issue.

Actual Results

Cache creation fails with following message "Cannot create cache as the path does not exist"

Anything else?

No response

repolevedavaj commented 1 year ago

@CSchulz can you send me your Jenkinsfile or at least a little bit more context on how you are using the plugin?

CSchulz commented 1 year ago

It is the same context like in #115. Tell me if you need more or what exactly and I will prepare it later. It is used within a node stage and normal stages.

CSchulz commented 1 year ago

Any update on this? Can I provide more details?

repolevedavaj commented 1 year ago
  1. Is this happening with the latest plugin release too?
  2. If you run something like sh 'ls -l .angular', the folder exists?
viceice commented 1 year ago

Seeing same when using full path:

[Cache for /tmp/.cache/nuget with id 0452b2873148fc3ca2edd47fcf313dd6] Cannot create cache as the path does not exist
[Pipeline] // cache
[Pipeline] sh
+ ls -la /tmp/.cache/nuget
total 0
drwxr-xr-x 6 root root 132 Jan  9 09:43 .
drwxr-xr-x 3 root root  19 Jan  9 09:43 ..
drwxr-xr-x 3 root root  19 Jan  9 09:43 mcmaster.extensions.commandlineutils
drwxr-xr-x 3 root root  19 Jan  9 09:43 minver
drwxr-xr-x 3 root root  19 Jan  9 09:43 system.componentmodel.annotations
drwxr-xr-x 3 root root  19 Jan  9 09:43 system.io.packaging
cache(maxCacheSize: 250, defaultBranch: 'main', caches: [arbitraryFileCache(path: '/tmp/.cache/nuget', cacheValidityDecidingFile: '**/packages.lock.json', compressionMethod: 'TAR_ZSTD')]) {
  sh 'dotnet restore --locked-mode'
}

sh 'ls -la /tmp/.cache/nuget'

NUGET_PACKAGES="/tmp/.cache/nuget" is set via stage environment

repolevedavaj commented 1 year ago

@viceice is this always happening for if you use a absolute path?

viceice commented 1 year ago

at least in that pipeline, yes. it's running inside a docker container. using a path inside the repo works as expected. but I explicit want it outside, so it's discarded after destroy of the docker container.

repolevedavaj commented 1 year ago

OK I will try it again using a Docker container

repolevedavaj commented 1 year ago

I can now finally reproduce this if I run the pipeline inside a docker container:

pipeline {
   agent {
      docker { image '...' }
   }
   stages {
      stage(...) {
         steps {
            cache(maxCacheSize: ..., caches: [...]) {
                ...
            }
         }
      }
   }
}

It seems that the workspace is in this case just mounted into the container. @viceice is this the way you are using docker?

viceice commented 1 year ago

yes, the workspace is checked out on host and then mounted to the container. that's the way the Jenkins docker agent works.

repolevedavaj commented 1 year ago

Right now I don't see an easy way of supporting paths outside the workspaces, but I will investigate a little bit more to see if the is a way of supporting this. Anyway, I think it would be helpful to at least notify the user in case a path outside the workspace is used.

viceice commented 1 year ago

i think I understand what's happening. the cache is only seeing the host filesystem, that's why I get the path not found. right?

so I'll switch back to paths inside workspace and just remove that in post stage step.

sadly there isn't a Jenkins agent option to clean the workspace after all steps are done, or i just can't find one. 🙃

repolevedavaj commented 1 year ago

Yes exactly. I will add an additional hint if using paths outside the workspace, but yeah I understand the inconvenience ;)

repolevedavaj commented 1 year ago

@CSchulz could this be related to your issue?

CSchulz commented 1 year ago
  1. Is this happening with the latest plugin release too?
  2. If you run something like sh 'ls -l .angular', the folder exists?

I am not able to update the plugin version for the moment, because we have fixed Jenkins version in Production. The folder is existing.

With absolute path ${workspace}/.angular it works.

repolevedavaj commented 1 year ago

@CSchulz what version are you using right now?

CSchulz commented 1 year ago

@CSchulz what version are you using right now?

Job Cacher plugin 291.v1b_2284da_5dd7

repolevedavaj commented 1 year ago

And which Jenkins version?

CSchulz commented 1 year ago

And which Jenkins version?

CloudBees Jenkins Enterprise 2.346.4.1-rolling

repolevedavaj commented 1 year ago

Thank you for the feedback. If you want, I can create you a build for this version so you can test it. What do you think?

CSchulz commented 1 year ago

In my opinion it is not required, because your suggested workaround is working for me. If this is working with the newest plugin version in all others cases as well, this issue can be closed.

dannykitchen commented 1 year ago

Is this possible at all. Use case. Container running rust. Need to cache the cargo folder that is on the container. As not in workspace currently not possible. Thanks

repolevedavaj commented 7 months ago

I close this issue as the original issue has been resolved. Please open a ticket in case you experience the same error in another situation.