Assess Artifactory bandwidth reduction options

MarkEWaite commented 1 year ago

Service(s)

Artifactory

Summary

JFrog has asked us to reduce the outbound bandwidth used by https://repo.jenkins-ci.org . One of the ideas being explored is to make several of the repository mirrors private. We need to test that by announcing and executing a series of time limited tests that temporarily make the repository mirrors private and assess the impact on Jenkins developers.

The proposed sequence of repositories to make private include:

[ ] jgit-cache
[ ] npm-cache
[ ] maven-repo1
[ ] jcenter-cache
[ ] nodejs-dist-cache
[ ] maven-repo1-cache

The repositories in the list include a mix of large and small repositories with some that are known to be used for Jenkins development and others that are not as clear that they are used for Jenkins development.

Implementation plan

Announce the series of functionality reduction tests with each lasting for a relatively brief period (1 hour). Announce in

[ ] developers list
[ ] one or more chat channels

During the functionality reduction tests, we will specifically assess impact on

[ ] ci.jenkins.io jobs
[ ] infra.ci.jenkins.io jobs
[ ] developer environments

timja commented 1 year ago

Nodejs dist can’t be made private as it will break builds as far as I know.

npm is likely the same depending on if it’s caching the npm binary or dependencies, (binary is the one that would cause problems)

dduportal commented 1 year ago

Nodejs dist can’t be made private as it will break builds as far as I know.

npm is likely the same depending on if it’s caching the npm binary or dependencies, (binary is the one that would cause problems)

What are the usages of NPM/NodeJS for artifactory? It's not really known by infra team

timja commented 1 year ago

Every CI build for a plugin that uses node / npm and core will download it from the mirror: https://github.com/jenkinsci/plugin-pom/blob/d749834a565493e17df4598e4d394749ff51dd00/pom.xml#L131-L132

dduportal commented 1 year ago

Every CI build for a plugin that uses node / npm and core will download it from the mirror: https://github.com/jenkinsci/plugin-pom/blob/d749834a565493e17df4598e4d394749ff51dd00/pom.xml#L131-L132

Thanks for the explanation @timja !

As pointed by @lemeurherve, we should check to put this usage under ACP, like for Maven, to decrease the amount of data downloaded from JFrog

dduportal commented 1 year ago

Timing proposal discussed during the weekly meeting:

jgit brownout: Friday 2 June
maven-repo1 brownout: 5 or 6 June

dduportal commented 1 year ago

Update: moved the brownout tests.

Proposal (ping @MarkEWaite @smerle33 @lemeurherve for voting +1 or -1 to this message)

jgit Thursday 8 June at 12h30 UTC (@MarkEWaite we can do it as we have a 1:1 at this time)
maven-repo1 Monday 12 June 12h30 if (and only if) the jgit test is successful.

If we got majority of vote, I'll open a status page + will send an email to the developers

smerle33 commented 1 year ago

+1 if needed as a voting and not an emoji

dduportal commented 1 year ago

Thanks folks! I need review (and approval if ok) on https://github.com/jenkins-infra/status/pull/310 then, before I send the email

MarkEWaite commented 1 year ago

+1 from me

dduportal commented 1 year ago

email thread for jgit brownout: https://groups.google.com/g/jenkinsci-dev

dduportal commented 1 year ago

Update:

Brownout is finished (closed email thread and status.jenkins.io)
@MarkEWaite will summarize our results

MarkEWaite commented 1 year ago

@jtnord suggested an alternate approach that can reduce bandwidth use without requiring changes to new releases of pom files and updates to those pom files. The proposal does not implement all the requested changes, but it implements the most significant bandwidth reductions.

Password protect our cache of Maven central

The Maven root pom defines Maven central as a repository. We cache Maven central on https;//repo.jenkins-ci.org under the virtual repository https;//repo.jenkins-ci.org/public . If we password protect that cached copy of Maven central, then Jenkins core and Jenkins plugin builds will automatically fall back to request the artifacts from Maven central directly.

Testing password protection of our cache of Maven central

@jtnord noted that we can approximate a password protected Maven central cache with the mirrorsOf configuration in the settings.xml file of the current user. Specifically, I've added the following entries to my ~/.m2/settings.xml and confirmed that I can build Jenkins core and more than 20 Jenkins plugins:

  <mirrors>
    <mirror>
      <id>repo.jenkins-ci.org</id>
      <url>https://repo.jenkins-ci.org/releases/</url>
      <mirrorOf>!repo1,!central,!incrementals,!jcenter-cache.jenkins-ci.org,!jgit-repository,*</mirrorOf>
    </mirror>
  </mirrors>

That mirror definition declares that https://repo.jenkins-ci.org/releases is a mirror of all repositories except repositories with the identifiers repo1, central, incrementals, jcenter-cache.jenkins-ci.org, and jgit-repository.

I included a Jenkins profile in my ~/.m2/settings.xml file to define those repositories:

  <profiles>
    <profile>
      <id>jenkins</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/releases/</url>
        </repository>
        <repository>
          <id>jcenter-cache.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/jcenter-cache/</url>
        </repository>
        <repository>
          <id>jgit-repository</id>
          <url>https://repo.eclipse.org/content/groups/releases/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/releases/</url>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>

When I build Jenkins core and Jenkins plugins, it downloads the Jenkins release artifacts (like jenkins.war) from the releases repository and it downloads artifacts that are not in the Jenkins release artifacts repository from the jcenter-cache repository (for jcrypt 1.0.0), the Eclipse JGit repository (for JGit), or Maven central (the fallback). If we moved jcrypt 1.0.0 from jcenter-cache to the Jenkins releases repository, we could remove it from that list. As far as I can tell, it is the only component pulled from jcenter-cache by Jenkins core or the 20+ plugins that I tested.

The changes that I was testing are broader than what was proposed by @jtnord, because they limit my access to only the releases repository, not to the other 10+ repositories that are included in the Jenkins public virtual repository.

This needs further discussion after I return from vacation, but it looks quite promising that we might be able to make significant and immediate progress with a single change.

This won't block access to other cached repositories in the Jenkins public virtual repository, but it would block access to most heavily used cache, Maven central.

dduportal commented 1 year ago

Following the recommendations of @jtnord and @daniel-beck, we have to plan a 1 hour "brownout" which goal would be to test the removal of the Maven Central mirrored repository from repo.jenkins-ci.org/public (instead of adding user password).

The following steps would be needed:

Before the brownout

[x] ASAP: Update the (ci.jenkins.io's) settings.xml to exclude the ID central from the mirrorOf directive to let Maven directly download artefacts from central when not found in repo.jenkins-ci.org/public - https://github.com/jenkins-infra/jenkins-infra/pull/2992
- Pros: Reproducible behavior (between ci.jenkins.io with ACP, Plugin CD releases in GitHub Actions, and contributors laptops) of the dependency resolutions
- Cons: No caching of the Maven central's artefacts in ci.jenkins.io builds (in the future we can setup ACP to manage this if the build time is really impacted)
[x] Decide and Announce the brownout date and time
- 2023-08-02 at 14h30 UTC
- https://github.com/jenkins-infra/status/pull/358 => status.jenkins.io up to date
- https://groups.google.com/g/jenkinsci-dev/c/82kPS0g1jDQ

During the brownout

[x] Empty the ACP cache on the 6 instances (2 replicas per cloud) by moving the persistent cache to a temp dir (in the same PVC: moving inodes is faster than deleting and we want to revert the cache back after). Worst case: rm -rf <nginx cahce dir> on each
- [ ] Script to be done for moving cache to a temp dir in the PVC
[x] Remove the no-transfer-progress option in https://github.com/jenkins-infra/pipeline-library/blob/8ed423572b7ade54219dcc6472a120a82c5c1145/vars/infra.groovy#L164 to allow us diagnosing the artefacts downloads in build logs
- https://github.com/jenkins-infra/pipeline-library/commit/67463678e9ba6375db0d442890a9a3df9190affe
[x] In Artifactory (repo.jenkins-ci.org) an administrator removes the Maven central repository from the public Virtual Repository
[ ] Test a first set of builds to study the behavior (and keep this build forever with a modified description pointing to the issue here):
- [x] Master CI build of a plugin: git client plugin
- [x] PR with incremental for plugin: embeddable build status plugin pull requests
- [x] Release of a plugin:
- (failed, see notes) ~MRP (manual) - poll scm plugin~
- [x] CD (in GHA) - embeddable build status plugin
- [x] Build in infra.ci - https://infra.ci.jenkins.io/job/reports/job/backend-extension-indexer/job/master/55/console
- [x] Test a BOM build - https://ci.jenkins.io/job/Tools/job/bom/job/master/1937/console
- [x] Test an ATH build - https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/896/console

End of brownout

[x] Roll back the ACP cache
(proposal to keep this setting until next brownout) ~Add back the no-transfer-progress option~
[x] Close operation
- https://github.com/jenkins-infra/status/pull/359
- https://groups.google.com/g/jenkinsci-dev/c/82kPS0g1jDQ/m/BXw32yeKBAAJ

jglick commented 1 year ago

Release of a plugin

Manually with MRP, or CD-enabled via GHA, or both?

we will only check the prep stage

I think you need a full-test since PCT runs will be running Maven with the tested plugin’s versioned POM, which may behave differently BOM builds.

dduportal commented 1 year ago

Release of a plugin

Manually with MRP, or CD-enabled via GHA, or both?

we will only check the prep stage

I think you need a full-test since PCT runs will be running Maven with the tested plugin’s versioned POM, which may behave differently BOM builds.

Both releases methods
True, thanks for the reminder

=> I've updated my comment above. Thanks!

dduportal commented 1 year ago

Outside the brownout plan, here are my notes from yesterday exchange with @jtnord @MarkEWaite @daniel-beck @lemeurherve @smerle33

Reminder by @jtnord that putting a repository list in pom (or parent pom) is discouraged: https://blog.sonatype.com/2009/02/why-putting-repositories-in-your-poms-is-a-bad-idea/
If the brownout shows that builds are not failing and are downloading central artefacts, we'll want to persist this setup.
- We might want to plan a 1 or 2 day brownout before planning the definitive change, to ensure we cover more cases
- To be discussed with Jfrog if it fits their requests
One of the core issues left is that most of the mirrored repositories should be in central (so downloads would be done through central instead of Jenkins Artifactory)
- jgit is an ideal candidate as it seems to have been included in central: we should be able to plan to remove it from our mirrors (ref. https://search.maven.org/artifact/org.eclipse.jgit/org.eclipse.jgit-parent/6.6.0.202305301015-r/pom)
- Would we have other mirrors that are now stored in central?
- We have at least 1 mirrored repository (javanet2 IIUC) which upstream is gone. Infra team want to ask JFrog for help in converting these repositories into hosted repositories.
- Do we have other (cc @daniel-beck) repositories in this case (upstream gone)?
Another potential question: we could start using ACP in the GitHub Actions builds (CD process)
- Technically, the infra team can restrict ACP access from only the GHA IPs + password
- Benefit would be a common behavior of CI builds and plugin release builds
- Risk is the additional work (by humans) + workload (on ACP)
Another long term question raised by @jtnord: publish everything to central?

jglick commented 1 year ago

we could start using ACP in the GitHub Actions builds (CD process)

We could, though these should not account for that much load to begin with. Is it worth the bother?

jglick commented 1 year ago

publish everything to central?

FTR OSSRH-93431

jglick commented 1 year ago

You did not do anything yet, did you? https://github.com/jenkinsci/bom/pull/2297/checks?check_run_id=15440825146

Could not transfer artifact com.github.jnr:jnr-x86asm:jar:1.0.2 from/to do-proxy (https://repo.do.jenkins.io/public/): status code: 403, reason phrase: Forbidden (403)

MarkEWaite commented 1 year ago

No changes have been applied to artifactory or to the proxies related to the bandwidth reduction project as far as I know. I may have missed a change to the artifact caching proxy that is preparatory for next week's brownout.

@lemeurherve would know for sure

jglick commented 1 year ago

Seems to have a transient issue.

dduportal commented 1 year ago

You did not do anything yet, did you? https://github.com/jenkinsci/bom/pull/2297/checks?check_run_id=15440825146

Could not transfer artifact com.github.jnr:jnr-x86asm:jar:1.0.2 from/to do-proxy (https://repo.do.jenkins.io/public/): status code: 403, reason phrase: Forbidden (403)

The job is HTTP/404 in Jenkins as the PR was merged so I can't analyse.

But this message is unexpected: repo.do.jenkins.io is the ACP instance running inside DigitalOcean Cloud. BOM build are expected to run in AWS which makes me worry a bit. Do you remember if it was the prep stage which failed?

jglick commented 1 year ago

Yes, just prep, since this PR was not full-test.

dduportal commented 1 year ago

Announce done: next step: update settings.xml and prepare ACP cache cleanup script

dduportal commented 1 year ago

Update: thanks @lemeurherve for https://github.com/jenkins-infra/kubernetes-management/pull/4217 (which is needed to test https://infra.ci.jenkins.io/job/reports/job/backend-extension-indexer/job/master/55/console

dduportal commented 1 year ago

First brownout finished, we'll share feedbacks here

MarkEWaite commented 1 year ago

The good news

Brownout proceeded as outlined in the earlier comment https://github.com/jenkins-infra/helpdesk/issues/3599#issuecomment-1653881431.

The artifact caching proxy contents were emptied (moved aside) and the proxies were restarted.
The two repo-1 caches were removed from the public artifact repository on repo.jenkins-ci.org.
The --no-transfer-progress argument was removed from the buildPlugin script so that we could see download sources and progress.
Successful CI builds were run with the git client plugin and the embeddable build status plugin using Linux and Windows container agents with Java 11 and Java 17. Logs for those builds confirmed that Maven central was used for artifacts provided by Maven central and that the artifact caching proxy was used for artifacts provided by the Jenkins project.

The bad news

The plugin bill of materials build failed when it was interrupted by a new commit. The interruption was unrelated to the brownout. There was an unexpected error message that needs more investigation. The message was:

Click here to expand the log

````shell 09:32:27 + mvn -f sample-plugin clean package -Dset.changelist -P2.387.x 09:32:29 [INFO] Setting: -Dchangelist=2283.v83b_709b_0b_8b_8 -DscmTag=83b709b0b8b8e02e6f4199f8cabb3342df5b7632 09:32:29 [INFO] No information available to set -DgitHubRepo 09:32:29 [INFO] Scanning for projects... 09:32:32 [INFO] Artifact org.jenkins-ci.tools:maven-hpi-plugin:pom:3.47 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [do-proxy-incrementals (https://repo.do.jenkins.io/incrementals/, default, releases), central (https://repo.maven.apache.org/maven2, default, releases)] 09:32:32 [WARNING] The POM for org.jenkins-ci.tools:maven-hpi-plugin:jar:3.47 is missing, no dependency information available 09:32:32 [INFO] Artifact org.jenkins-ci.tools:maven-hpi-plugin:jar:3.47 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [do-proxy-incrementals (https://repo.do.jenkins.io/incrementals/, default, releases), central (https://repo.maven.apache.org/maven2, default, releases)] 09:32:32 [WARNING] Failed to build parent project for io.jenkins.tools.bom:sample:hpi:2283.v83b_709b_0b_8b_8 09:32:33 [INFO] 09:32:33 [INFO] --------------------< io.jenkins.tools.bom:sample >--------------------- 09:32:33 [INFO] Building sample 2283.v83b_709b_0b_8b_8 09:32:33 [INFO] from pom.xml 09:32:33 [INFO] --------------------------------[ hpi ]--------------------------------- 09:32:35 Superseded by [#1938](https://ci.jenkins.io/job/Tools/job/bom/job/master/1938/) 09:32:35 Sending interrupt signal to process 09:32:38 script returned exit code 143 ````

Unexpected messages were also seen in local builds that match the messages in the bom build. Needs more investigation.

Local release of pollscm plugin with mvn release:prepare release:perform failed unexpectedly for Mark Waite. The hpi file was delivered to repo.jenkins-ci.org but the pom file was not delivered there. We're not sure if the failure is due to a local configuration problem on Mark Waite's computer or some other issue. The build message was:

Click here to expand the log

```shell [INFO] [INFO] --- install:3.1.1:install (default-install) @ pollscm --- [INFO] [INFO] Installing /home/mwaite/hub/my-plugins/pollscm-plugin/target/checkout/pom.xml to /home/mwaite/.m2/repository/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.pom [INFO] [INFO] Installing /home/mwaite/hub/my-plugins/pollscm-plugin/target/checkout/target/pollscm.hpi to /home/mwaite/.m2/repository/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.hpi [INFO] [INFO] Installing /home/mwaite/hub/my-plugins/pollscm-plugin/target/checkout/target/pollscm.jar to /home/mwaite/.m2/repository/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.jar [INFO] [INFO] Installing /home/mwaite/hub/my-plugins/pollscm-plugin/target/checkout/target/pollscm-sources.jar to /home/mwaite/.m2/repository/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-sources.jar [INFO] [INFO] Installing /home/mwaite/hub/my-plugins/pollscm-plugin/target/checkout/target/pollscm-javadoc.jar to /home/mwaite/.m2/repository/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-javadoc.jar [INFO] [INFO] [INFO] [INFO] --- deploy:3.1.1:deploy (default-deploy) @ pollscm --- [INFO] Uploading to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.pom [INFO] Progress (1): 3.3 kB [INFO] [WARNING] Failed to upload checksum to org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.pom.sha1 [INFO] org.apache.http.client.HttpResponseException: status code: 403, reason phrase: (403) [INFO] at org.eclipse.aether.transport.http.HttpTransporter.handleStatus (HttpTransporter.java:552) [INFO] at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:421) [INFO] at org.eclipse.aether.transport.http.HttpTransporter.implPut (HttpTransporter.java:402) [INFO] at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put (AbstractTransporter.java:107) [INFO] at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:608) [INFO] at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:591) [INFO] at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:565) [INFO] at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:414) [INFO] at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:297) [INFO] at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:270) [INFO] at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:201) [INFO] at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:392) [INFO] at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:139) [INFO] at org.apache.maven.plugins.deploy.DeployMojo.execute (DeployMojo.java:203) [INFO] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162) [INFO] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39) [INFO] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159) [INFO] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105) [INFO] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73) [INFO] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53) [INFO] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118) [INFO] at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261) [INFO] at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173) [INFO] at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101) [INFO] at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906) [INFO] at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283) [INFO] at org.apache.maven.cli.MavenCli.main (MavenCli.java:206) [INFO] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) [INFO] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) [INFO] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) [INFO] at java.lang.reflect.Method.invoke (Method.java:566) [INFO] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283) [INFO] at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226) [INFO] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407) [INFO] at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348) [INFO] [INFO] Uploaded to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.pom (3.3 kB at 3.3 kB/s) [INFO] Uploading to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.hpi [INFO] Uploading to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.jar [INFO] Uploading to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-sources.jar [INFO] Uploading to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-javadoc.jar [INFO] Progress (1): 11 kB [INFO] [INFO] Progress (1): 8.1 kB [INFO] Progress (2): 8.1 kB | 15 kB [INFO] Progress (3): 8.1 kB | 15 kB | 33/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 66/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 98/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 131/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 164/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 197/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 229/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 262/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 295/324 kB [INFO] Progress (3): 8.1 kB | 15 kB | 324 kB [INFO] [INFO] Uploaded to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5.hpi (15 kB at 15 kB/s) [INFO] Uploaded to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-javadoc.jar (324 kB at 322 kB/s) [INFO] Uploaded to maven.jenkins-ci.org: https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pollscm/1.5/pollscm-1.5-sources.jar (8.1 kB at 6.8 kB/s) [INFO] [INFO] ------------------------------------------------------------------------ [INFO] [INFO] BUILD FAILURE [INFO] [INFO] ------------------------------------------------------------------------ [INFO] [INFO] Total time: 8.026 s [INFO] [INFO] Finished at: 2023-08-02T09:20:48-06:00 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.1.1:deploy (default-deploy) on project pollscm: Failed to deploy artifacts: Could not transfer artifact org.jenkins-ci.plugins:pollscm:jar:1.5 from/to maven.jenkins-ci.org (https://repo.jenkins-ci.org/releases/): status code: 403, reason phrase: (403) -> [Help 1] [INFO] [ERROR] [INFO] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [INFO] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [INFO] [ERROR] [INFO] [ERROR] For more information about the errors and possible solutions, please read the following articles: [INFO] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 30.130 s [INFO] Finished at: 2023-08-02T09:20:48-06:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.1:perform (default-cli) on project pollscm: Maven execution failed, exit code: 1 -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException ````

jglick commented 1 year ago

Can https://github.com/jenkins-infra/pipeline-library/commit/67463678e9ba6375db0d442890a9a3df9190affe be reverted? It is making build logs quite verbose.

dduportal commented 1 year ago

Can jenkins-infra/pipeline-library@6746367 be reverted? It is making build logs quite verbose.

Done by https://github.com/jenkins-infra/pipeline-library/commit/1c849640d0fe9f8ae5095d1b08e8360efee8815d + cleared up the library cache on ci.jenkins.io.

MarkEWaite commented 1 year ago

The backend extension indexer job appears to have stopped working about the same time we performed the brownout. We'll need to investigate further to understand the root cause of the failure.

dduportal commented 1 year ago

Update: as part of https://github.com/jenkins-infra/helpdesk/issues/3737 which reports the dreaded error

Unresolveable build extension: Plugin org.jenkins-ci.tools:maven-hpi-plugin:XXX or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.jenkins-ci.tools:maven-hpi-plugin:jar:XXX (absent): Could not find artifact org.jenkins-ci.tools:maven-hpi-plugin:jar:XXX in <some artifact caching proxy ID> (<some artifact caching proxy URL>) -> [Help 2]

when building on ci.jenkins.io. But:

Do not use any settings.xml (contributor laptop for instance) then it works
Specify an explicit pluginRepository pointing to repo.jenkins-ci.org/public in the failing project's pom.xml then it works

It looks like that using a settings.xml (like ci.jenkins.io agents) with a mirrorOf directive (purpose of catching all donwload requests and send them to caching proy) does not behave as expected with the Maven plugin resolution.

But this file can be tuned to auto-enable a custom profile which defines the proper plugin repository resolution system (in the context of ci.jenkins.io). I was successful in a local attempt with the following block added to my settings.xml (without it, I reproduce the error):

<profiles>
    <profile>
      <id>jenkins-infra-plugin-repositories</id>
      <pluginRepositories>
        <pluginRepository>
          <id>repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/public/</url>
        </pluginRepository>
        <pluginRepository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>incrementals</id>
          <url>https://repo.jenkins-ci.org/incrementals/</url>
        </pluginRepository>
        <pluginRepository>
          <id>central</id>
          <url>https://repo.maven.apache.org/maven2</url>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>jenkins-infra-plugin-repositories</activeProfile>
  </activeProfiles>

My proposal is to add this code block (only for pluginRepository as we have not any other error) in the existing settings.xml file on ci.jenkins.io.

WDYT @MarkEWaite @jnord @basil @daniel-beck ? Does it make sense and is it a valid assessment considering we want to remove the "Maven Central" mirrors from repo.jenkins-ci.org?

basil commented 1 year ago

I've seen that error a few times and chased it away by disabling the artifact caching proxy. I suspect the artifact caching proxy is still caching some results from the last "experiment" and this problem would go away if the cache was cleared.

dduportal commented 1 year ago

I've seen that error a few times and chased it away by disabling the artifact caching proxy. I suspect the artifact caching proxy is still caching some results from the last "experiment" and this problem would go away if the cache was cleared.

That was my initial though but I tried without cache and still had the error. However, cleaning up the cache at all is on the item list for the upcoming brownout!

The cache cleanup remove the following error though: [INFO] Artifact <some dependency path> is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [<list of repositories which order changes between different builds>]

basil commented 1 year ago

The cache cleanup remove the following error though: [INFO] Artifact <some dependency path> is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [<list of repositories which order changes between different builds>]

Ah yes, that is the error I was thinking of, not the "Unresolveable build extension" one.

daniel-beck commented 1 year ago

Profile seems like a pragmatic solution.

Curious why this doesn't work though. Are plugin repos special wrt mirrors? Are we not mirroring plugins' special meta-metadata.xml file?

dduportal commented 1 year ago

Profile seems like a pragmatic solution.

Thanks for the quick reply folks!

Are we not mirroring plugins' special meta-metadata.xml file?

I did not know about this file. The ACP configuration does not cache the maven-metadata.xml and directly serves it from Artifactory (https://github.com/jenkins-infra/helm-charts/blob/a38df338a0801899d7f54b78a62793042c5dc62d/charts/artifact-caching-proxy/templates/nginx-proxy-configmap.yaml#L57-L60) but for meta-metadata.xml I believe it should be present. Worth digging this.

Curious why this doesn't work though. Are plugin repos special wrt mirrors?

Same here I don't understand the root cause and it is a bit frustrating. I have a gut feeling it is related to the mirrorOf directive

daniel-beck commented 1 year ago

but for meta-metadata.xml I believe it should be present. Worth digging this.

Oops, typo. I meant maven-metadata.xml, but it's on the group level, not (just) the artifact level: https://repo.jenkins-ci.org/releases/org/jenkins-ci/tools/

Perhaps that makes a difference?

dduportal commented 1 year ago

In relation with the bandwidth issue, here is the "next steps" proposal:

Thursday 31 August 2023: deployment of the new settings.xml with the proposed plugin repositories profile (targets: ci.jenkins.io and infra.ci.jenkins.io)
Monday 4 September 2023: brownout of 4 hours (noon UTC until 16:00 UTC) with no more Apache Central
- Currently checking with stakeholders
Thursday 7 September: if the brownout is a success, we definitively remove Apache Central mirrors from Artifactory

dduportal commented 1 year ago

Update:

New settings.xml file deployed to ci.jenkins.io and infra.ci.jenkins.io.
Opened a draft PR in https://github.com/jenkinsci/nexus-platform-plugin/pull/278 to validate the change
Starting a bom and ATH build to verify the behavior

dduportal commented 1 year ago

Update: proposed timeline for the next brownout https://github.com/jenkins-infra/status/pull/370

dduportal commented 1 year ago

Brownout started:

Removed the 2 *repo1* mirrors from repo.jenkins-ci.org/public
Cleaned up the cache of the 6 ACPs replicas (2 per clouds) and restarted processes to avoid serving in-memory cache
Announced in usual Im channels
status.jenkins.io updated
added a messae on ci.jenkins.io

Next steps: testing builds

dduportal commented 1 year ago

Brownout is finished: closed status.jenkins.io and sent an email on the mailing list.

TL;DR; results are really good, we only have one last build issue in https://github.com/jenkinsci/maven-hpi-plugin/pull/529#issuecomment-1705463280 but not blocking the planning as it is only a matter of Integration tests and settings.xml

(edit) Detailled reports of what was done and tested during the brownout:

Artifactory: Removed repo1 and maven-repo1 mirror repositories from the public virtual repository
"ACP" (Artifact Caching Proxy) inside the Jenkins infrastructure: cleaned up the cache for each replica
- 6 replicas: 2 per cloud provider (eks-public in AWS EKS, doks-public in Digital Ocean and publick8s in AKS)
- On each pod, ran the rm -rf /data/nginx-cache/* then delete the pod to force the nginx process to be restarted
Tested the following jobs on ci.jenkins.io:
- pipeline-library master branch
- No error or warnings not already present on the previous builds
- BOM master branch
- No error or warnings not already present on the previous builds
- ATH master branch
- No error or warnings not already present on the previous builds
- Plugin master branch (on jenkins-infra-test-plugin)
- No error, no warnings
- Plugin PR (on nexus-platform-plugin)
- Saw the following warning but it does not seem to change the build or test behavior:
```
[WARNING] Error resolving project artifact: The following artifacts could not be resolved: com.sun:tools:pom:1.8.0 (absent): Could not find artifact com.sun:tools:pom:1.8.0 in azure-proxy (https://repo.azure.jenkins.io/public/) for project com.sun:tools:jar:1.8.0
```
- maven-hpi-plugin (the most trickier). As per https://github.com/jenkinsci/maven-hpi-plugin/pull/529, the integration tests are broken since https://github.com/jenkins-infra/jenkins-infra/pull/3041, but no new error/warning due to the brownout actions
Tested the following jobs on trusted.ci.jenkins.io:
- update center
- RPU
- javadoc
- crawler
- jenkins.io
- core-tags-lib-generator
Artifactory: Added back repo1 and maven-repo1 mirror repositories in the public virtual repository
"ACP" (Artifact Caching Proxy) inside the Jenkins infrastructure: cleaned up the cache for each replica

dduportal commented 1 year ago

Update: next steps:

As discussed during the 2023-09-05 weekly infra team meeting there is no objection or blocker against definitively removing the Apache Central repositories (e.g. persisting the brownout setting)
JFrog confirmed that it looks good to them
Proposed deadline: Thursday September 7 2023 (around 08:00am UTC)
⚠️ The result on the effective outbound bandwidth might not be immediate as we'll have to clean the ACP cache, which will give a burst in download until is it repopulated (usually takes 5 to 7 days)
Communication to end users:
- Blog post in jenkins.io - https://github.com/jenkins-infra/jenkins.io/pull/6658/files
- Mailing list message - https://groups.google.com/g/jenkinsci-dev/c/NBs1FXCiaSM
- Usual IM messages
Operation should be:
- Artifactory: Remove repo1 and maven-repo1 mirror repositories from the public virtual repository
- "ACP" (Artifact Caching Proxy) inside the Jenkins infrastructure: cleaned up the cache for each replica
- 6 replicas: 2 per cloud provider (eks-public in AWS EKS, doks-public in Digital Ocean and publick8s in AKS)
- On each pod, ran the rm -rf /data/nginx-cache/* then delete the pod to force the nginx process to be restarted
- Set both repo1 and maven-repo1 as private mirrors

dduportal commented 1 year ago

Work on https://github.com/jenkinsci/maven-hpi-plugin/pull/537 is finished and merged thanks to the help of @basil .

It is a fix to allow running Integrations Tests by opt-ing out of ACP for this project (only).

Background work is needed for a clear fix (issue to track this in https://github.com/jenkinsci/maven-hpi-plugin/issues/541).

Next step to close this issue: let's wait from @MarkEWaite 's analysis of the logs provided by JFrog and confirm with them that the new bandwitdh usage is fine for them.

MarkEWaite commented 1 year ago

Done. Log file format changed and we've decided to not spend the effort to adapt to the changed log file format. Thanks to all!

jenkins-infra / helpdesk