Recent update to 252.v3b_9c37dfa_a_33 caused Amazon S3 storage to stop working

stronk7 commented 2 years ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.346.1 OS: Linux - 5.4.196-108.356.amzn2.x86_64 --- ace-editor:1.1 all-changes:1.5 ansicolor:1.0.1 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.215-343.v953e693f0e86 aws-java-sdk-cloudformation:1.12.215-343.v953e693f0e86 aws-java-sdk-codebuild:1.12.215-343.v953e693f0e86 aws-java-sdk-ec2:1.12.215-343.v953e693f0e86 aws-java-sdk-ecr:1.12.215-343.v953e693f0e86 aws-java-sdk-ecs:1.12.215-343.v953e693f0e86 aws-java-sdk-elasticbeanstalk:1.12.215-343.v953e693f0e86 aws-java-sdk-iam:1.12.215-343.v953e693f0e86 aws-java-sdk-logs:1.12.215-343.v953e693f0e86 aws-java-sdk-minimal:1.12.215-343.v953e693f0e86 aws-java-sdk-sqs:1.12.215-343.v953e693f0e86 aws-java-sdk-ssm:1.12.215-343.v953e693f0e86 bootstrap4-api:4.6.0-5 bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-failure-analyzer:2.2.1 build-timeout:1.21 build-user-vars-plugin:1.8 build-with-parameters:1.6 caffeine-api:2.9.3-65.v6a_47d0f4d1fe categorized-view:1.12 checks-api:1.7.4 cloudbees-folder:6.729.v2b_9d1a_74d673 cobertura:1.17 code-coverage-api:2.0.4 collapsing-console-sections:1.8.0 command-launcher:84.v4a_97f2027398 conditional-buildstep:1.4.2 copyartifact:1.46.4 credentials:1129.vef26f5df883c credentials-binding:523.vd859a_4b_122e6 dashboard-view:2.432.va_712ce35862d data-tables-api:1.12.1-1 display-upstream-changes:0.3.2 display-url-api:2.3.6 docker-build-publish:1.3.3 docker-commons:1.19 docker-workflow:1.28 downstream-buildview:1.9 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 embeddable-build-status:2.0.4 extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:191.v363d0d1efdf8 extra-columns:1.25 font-awesome-api:6.1.1-1 forensics-api:1.15.1 git:4.11.3 git-client:3.11.0 git-parameter:0.9.17 git-server:1.11 google-login:1.6 greenballs:1.15.1 handlebars:3.0.8 jackson2-api:2.13.3-285.vc03c0256d517 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.5 jira:3.7.1 jnr-posix-api:3.1.7-3 job-import-plugin:3.5 jobConfigHistory:1148.v8607da_ef251e jobcacher:252.v3b_9c37dfa_a_33 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 junit-attachments:101.v82f494a_00e9e ldap:2.10 leastload:3.0.0 locale:144.v1a_998824ddb_3 lockable-resources:2.15 mailer:414.vcc4c33714601 matrix-auth:2.6.8 matrix-project:771.v574584b_39e60 maven-plugin:3.19 metrics:4.1.6.2 metrics-graphite:3.0.0 mina-sshd-api-common:2.8.0-21.v493b_6b_db_22c6 mina-sshd-api-core:2.8.0-21.v493b_6b_db_22c6 momentjs:1.1.1 monitoring:1.91.0 next-executions:1.0.15 nodelabelparameter:1.11.0 notification:1.15 pam-auth:1.8 parameterized-trigger:2.44 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2097.v33db_b_de764b_e pipeline-model-definition:2.2097.v33db_b_de764b_e pipeline-model-extensions:2.2097.v33db_b_de764b_e pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e pipeline-stage-view:2.24 plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.5-2 rebuild:1.34 resource-disposer:0.19 role-strategy:488.v0634ce149b_8c run-condition:1.5 s3:0.12.1 saml:2.298.vc7a_2b_3958628 scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 sectioned-view:1.25 snakeyaml-api:1.30.1 ssh-credentials:277.v95c2fec1c047 sshd:3.242.va_db_9da_b_26a_c3 strict-crumb-issuer:2.1.0 structs:318.va_f3ccb_729b_71 swarm:3.33 systemloadaverage-monitor:1.2 template-project:1.5.2 thinBackup:1.10 throttle-concurrents:2.8 timestamper:1.18 token-macro:293.v283932a_0a_b_49 trilead-api:1.57.v6e90e07157e1 variant:1.4 view-job-filters:2.3 windows-slaves:1.8.1 workflow-aggregator:581.v0c46fa_697ffd workflow-api:1164.v760c223ddb_32 workflow-basic-steps:948.v2c72a_091b_b_68 workflow-cps:2725.v7b_c717eb_12ce workflow-durable-task-step:1146.v1a_d2e603f929 workflow-job:1186.v8def1a_5f3944 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:820.vd1a_6cc65ef33 ws-cleanup:0.42 ws-ws-replacement:1.0.1 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

We are running the controller using unmodified jenkins/jenkins:lts docker image and the agents are real iron running Ubuntu 20.04 LTS

Reproduction steps

With the jobcacher-plugin installed
And configured globally to use a valid, existing AWS bucket:
- Storage implementation: Amazon S3 storage
- Credentials: xxxx
- S3 Bucker Name: xxxx
- Region: eu-west-1

In a pipeline we use this to cache between runs a few small json files containing some timing information:

stage("Run Task") {
steps {
    wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'xterm']) {
        cache(
            maxCacheSize: 250,
            caches: [
               [$class: 'ArbitraryFileCache', excludes: '', includes: '*.json', path: "timing"]
            ]
        ) {
            .... task definition continues here

Expected Results

The jobs should be executed properly and end with success (given the tests pass, heh)

Actual Results

The execution doesn't happen, the task finishes immediately after the cache use, with some traces that gave me the clue to come to this plugin. (example failed job)

Anything else?

All the jobs were working perfectly until a few hours ago, when we performed the upgrade (from plugin 1.0, I think).
it has been checked that the bucket is up and contains files from executions 2-3 hours ago. For example:
```
2022-06-24 14:36        62684  s3://xxxx/xxxx/cache/4ad8aa3a3571ea912a6ec5ea5fdcc93c/default.json
```
Changing the storage implementation to Built-in Jenkins storage makes the jobs to be back to run.

stronk7 commented 2 years ago

Adding to the above:

Downgrading the plugin to 1.0 also makes everything to be back to work.

repolevedavaj commented 2 years ago

@stronk7 thank you for your report. I implemented a fix which is solving this. Can you validate that the fix is working for you too? You can install the plugin from here: https://ci.jenkins.io/job/Plugins/job/jobcacher-plugin/view/change-requests/job/PR-56/

stronk7 commented 2 years ago

Hi @repolevedavaj,

I've installed it from that run (BTW, it fails to download directly and instead had to download the zip containing all the artifacts). And now I'm getting some weird dependencies error (I saw that a few AWSxxxx plugins were also available to update, so I updated them too).

This is what I'm getting:

Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to re-enable these plugins.

Dependency errors:

Amazon Web Services SDK :: All (1.12.246-349.v96b_b_f7eb_a_c3c) Update required: Amazon Web Services SDK :: Elastic Beanstalk (aws-java-sdk-elasticbeanstalk 1.12.215-343.v953e693f0e86) to be updated to 1.12.246-349.v96b_b_f7eb_a_c3c or higher Some of the above failures also result in additional indirectly dependent plugins not being able to load.

Indirectly dependent plugins:

Job Cacher plugin (263.v2c288201052d) Failed to load: Amazon Web Services SDK :: All (aws-java-sdk 1.12.246-349.v96b_b_f7eb_a_c3c)

In the mean time, I'm going to try downgrading the AWS plugins to previous versions, what a coincidence!

Ciao :-)

stronk7 commented 2 years ago

Aha, for reference, that problem has been tracked @ https://issues.jenkins.io/browse/JENKINS-68834

I'm going to proceed by installing the elastic beanstalk from there. Let's see.

Ciao :-)

stronk7 commented 2 years ago

Ok, with the elastic beanstalk dependencies problem above solved... I've launched a few jobs and they seem to be working ok. Caches have been restored ok, let's see when the jobs end how they are backup.

Example job (I've launched a bunch): https://ci.moodle.org/job/W.02.01%20-%20Behat%20-%20Goutte%20+%20Postgres/1987/console

Ciao :-)

PS : And thanks for the promptly fix! PS2 (edited) : Forgot to comment, those runs are with Job Cacher plugin 263.v2c288201052d (from the PR job)

stronk7 commented 2 years ago

And the caches have been backup perfectly, too:

2022-06-25 08:03        35075  s3://xxxxx/xxxxx/cache/4ad8aa3a3571ea912a6ec5ea5fdcc93c/default.json

Yay! Ciao :-)

repolevedavaj commented 2 years ago

@stronk7 Thank you for your feedback. We did not upgrade any dependency since the last release so maybe there was another plugin update which has lead to this conflict. Anyway I am glad it worked, so I will merge and release it 👍

jenkinsci / jobcacher-plugin