ci.jenkins.io no longer publishes incrementals

MarkEWaite commented 2 years ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.319.1 OS: Linux - 5.4.0-1064-azure --- ace-editor:1.1 analysis-model-api:10.8.1 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.5 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:1.33 aws-java-sdk:1.12.101-300.vc09c7be9cb57 aws-java-sdk-cloudformation:1.12.101-300.vc09c7be9cb57 aws-java-sdk-codebuild:1.12.101-300.vc09c7be9cb57 aws-java-sdk-core:1.12.69 aws-java-sdk-ec2:1.12.101-300.vc09c7be9cb57 aws-java-sdk-ecr:1.12.101-300.vc09c7be9cb57 aws-java-sdk-ecs:1.12.101-300.vc09c7be9cb57 aws-java-sdk-elasticbeanstalk:1.12.101-300.vc09c7be9cb57 aws-java-sdk-iam:1.12.101-300.vc09c7be9cb57 aws-java-sdk-jmespath:1.12.69 aws-java-sdk-kms:1.12.69 aws-java-sdk-logs:1.12.101-300.vc09c7be9cb57 aws-java-sdk-minimal:1.12.101-300.vc09c7be9cb57 aws-java-sdk-s3:1.12.69 aws-java-sdk-ssm:1.12.101-300.vc09c7be9cb57 aws-java-sdk-sts:1.12.69 azure-container-agents:215.v9ca34d6f145b azure-credentials:198.vf9c2fdfde55c azure-sdk:70.v63f6a95999a7 azure-vm-agents:799.va4c741108611 basic-branch-build-strategies:1.3.2 beer:1.3 blueocean:1.25.2 blueocean-autofavorite:1.2.4 blueocean-bitbucket-pipeline:1.25.2 blueocean-commons:1.25.2 blueocean-config:1.25.2 blueocean-core-js:1.25.2 blueocean-dashboard:1.25.2 blueocean-display-url:2.4.1 blueocean-events:1.25.2 blueocean-git-pipeline:1.25.2 blueocean-github-pipeline:1.25.2 blueocean-i18n:1.25.2 blueocean-jira:1.25.2 blueocean-jwt:1.25.2 blueocean-personalization:1.25.2 blueocean-pipeline-api-impl:1.25.2 blueocean-pipeline-editor:1.25.2 blueocean-pipeline-scm-api:1.25.2 blueocean-rest:1.25.2 blueocean-rest-impl:1.25.2 blueocean-web:1.25.2 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-4 bouncycastle-api:2.25 branch-api:2.7.0 build-timeout:1.20 buildtriggerbadge:2.11 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloud-stats:0.27 cloudbees-bitbucket-branch-source:734.v2f848c5e6ea2 cloudbees-folder:6.17 code-coverage-api:2.0.4 command-launcher:1.6 conditional-buildstep:1.4.1 configuration-as-code:1.55 copyartifact:1.46.2 credentials:1055.v1346ba467ba1 credentials-binding:1.27 cvs:2.19 dark-theme:155.v497c78bbdbb3 data-tables-api:1.11.3-6 disable-github-multibranch-status:1.2 display-url-api:2.3.5 docker-commons:1.17 docker-workflow:1.26 durable-task:493.v195aefbb0ff2 ec2:1.66 echarts-api:5.2.2-2 embeddable-build-status:2.0.3 extended-read-permission:3.2 external-monitor-job:1.7 favorite:2.3.3 font-awesome-api:5.15.4-5 forensics-api:1.7.0 git:4.10.1 git-client:3.10.0 git-forensics:1.3.0 git-server:1.10 github:1.34.1 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.3 github-checks:1.0.14 groovy:2.4 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.28 http_request:1.12 jackson2-api:2.13.1-242.v1a45bad25ceb jacoco:3.3.0 javadoc:1.6 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.2 jira:3.6 jjwt-api:0.11.2-9.c8b45b8bb173 jobConfigHistory:2.30 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.53 junit-attachments:1.7 junit-realtime-test-reporter:72.v203572adb123 keyboard-shortcuts-plugin:1.4 kubernetes:1.31.1 kubernetes-client-api:5.10.1-171.vaa0774fb8c20 kubernetes-credentials:0.9.0 ldap:2.7 lockable-resources:2.13 mailer:1.34 mapdb-api:1.0.9.0 matrix-auth:3.0 matrix-project:1.19 maven-plugin:3.16 mercurial:2.16 metrics:4.0.2.8 momentjs:1.1.1 node-iterator-api:1.5.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.6.1 parallel-test-executor:1.13 parameterized-trigger:2.43 pipeline-build-step:2.15 pipeline-github-lib:1.0 pipeline-githubnotify-step:1.0.5 pipeline-graph-analysis:1.12 pipeline-graph-view:47.v91726240f7a6 pipeline-input-step:427.va6441fa17010 pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.20 pipeline-stage-step:2.5 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.20 pipeline-utility-steps:2.11.0 plain-credentials:1.7 plugin-util-api:2.8.0 popper-api:1.16.1-2 popper2-api:2.11.0-1 prism-api:1.25.0-1 pubsub-light:1.16 run-condition:1.5 scm-api:2.6.5 script-security:1118.vba21ca2e3286 snakeyaml-api:1.29.1 sse-gateway:1.24 ssh-agent:1.23 ssh-credentials:1.19 ssh-slaves:1.33.0 sshd:3.1.0 structs:308.v852b473a2b8c subversion:2.15.1 support-core:2.79 theme-manager:0.6 throttle-concurrents:2.5 timestamper:1.15 token-macro:267.vcdaea6462991 toolenv:1.2 translation:1.16 trilead-api:1.0.13 variant:1.4 warnings-ng:9.10.2 windows-azure-storage:365.vf41653c43b01 windows-slaves:1.8 workflow-aggregator:2.6 workflow-api:1108.v57edf648f5d4 workflow-basic-steps:2.24 workflow-cps:2648.va9433432b33c workflow-cps-global-lib:552.vd9cc05b8a2e1 workflow-durable-task-step:1107.v5dab75aaccbd workflow-job:1145.v7f2433caa07f workflow-multibranch:2.26 workflow-scm-step:2.13 workflow-step-api:615.vb09dac339255 workflow-support:804.vba10a18a1476 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 18.04 for controller and for agents

Reproduction steps

Enable incrementals for a Jenkins plugin
Build the Jenkins plugin on ci.jenkins.io with the buildPlugin() pipeline library
Configure a plugins.txt file in a Dockerfile to use the plugin that has been built. In my case, I used
```
git-client:incrementals;org.jenkins-ci.plugins;3.10.2-rc3007.0409ea_938a_3a_
```
Attempt to download the plugin with plugin installation manager

Expected Results

Plugin downloaded and available for use

Actual Results

Plugin download fails because the incremental build was not uploaded to repos.jenkins-ci.org.

Build log reports:

10:02:25  [Pipeline] {
10:02:25  [Pipeline] httpRequest
10:02:25  HttpMethod: POST
10:02:25  URL: https://incrementals.jenkins.io/
10:02:25  Content-Type: application/json
10:02:25  Authorization: *****
10:02:25  Sending request to url: https://incrementals.jenkins.io/
10:02:26  Response Code: HTTP/1.1 200 OK
10:02:26  Response: 
10:02:26  Skipping deployment as no artifacts were found with the expected path, typically due to a PR merge build not up to date with its base branch: https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/728/artifact/**/*0409ea938a3a*/*0409ea938a3a*/*zip*/archive.zip
10:02:26  
10:02:26  Success: Status code 200 is in the accepted range: 100:599
10:02:26  [Pipeline] }

Anything else?

Likely needs a change in the argument to the archiveArtifacts task in the pipeline library used on ci.jenkins.io.

jglick commented 2 years ago

https://github.com/jenkins-infra/incrementals-publisher/pull/20 should have fixed this. Not sure what is going on with the deployment. https://github.com/jenkins-infra/helm-charts/pull/57? https://github.com/jenkins-infra/helm-charts/pull/67? @timja or @jetersen do you know?

timja commented 2 years ago

This was reported elsewhere where I gave more detail but basically we're hit by https://issues.jenkins.io/browse/JENKINS-67474 which has caused complete blockage of infra builds preventing updates.

It's being worked on in:

Getting close atm I think, mostly @dduportal and @lemeurherve working on it

Current state from what I can tell is the parallelDockerUpdatecli is broken from pipeline-library, can be seen in build logs for https://github.com/jenkins-infra/docker-helmfile/pull/45

dduportal commented 2 years ago

For information, the branch master of this project is now building successfully on infra.ci.jenkins.io: https://infra.ci.jenkins.io/job/incrementals-publisher/job/master/ .

Last mile for us, before being able to check ci.jenkins.io is to ensure that our kubernetes management works as expected.

dduportal commented 2 years ago

Kubernetes management is now operational "again" and has detected the changes on the incremental application:

https://infra.ci.jenkins.io/job/helm-charts/job/main/2/console is now working as expected (updatecli process for detecting applications update and open PRs to update their helm charts - https://github.com/jenkins-infra/helm-charts/blob/main/Jenkinsfile_k8s)
https://github.com/jenkins-infra/helm-charts/pull/67 had been updated and builds successfully. Approval & merge in progress.
In a few minutes, https://github.com/jenkins-infra/kubernetes-management will automatically pick the change and update.

dduportal commented 2 years ago

One last mile was required: https://github.com/jenkins-infra/kubernetes-management/pull/1886.

The last known version of the incrementalm-publisher (v1.4.2) is deployed successfully:

Successfully pulled image "jenkinsciinfra/incrementals-publisher:v1.4.2" in 25.898692355s

=> Can you confirm that incremental publishing is working again as expected?

MarkEWaite commented 2 years ago

=> Can you confirm that incremental publishing is working again as expected?

https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/735/console confirms that incremental plugin publishing is working. The build results are deployed to repos.jenkins-ci.org.

I used git-client:incrementals;org.jenkins-ci.plugins;3.11.1-rc3016.2b_fb_197d7b_9c in my plugins.txt file and the plugin installation manager downloaded that plugin version from repos.jenkins-ci.org.

jenkinsci / incrementals-tools