[INFRA-1139] Mirror of node dist and npm registry - and usage of it in plugin-pom

[jenkins-infra-bot]

from the "misuse of funds" department, plugin builds fail sometimes due to nodejs.org/dist or npm registry being unavailable. 

One recent case is: https://github.com/nodejs/nodejs.org/issues/1191

In this case it was the distribution of binaries (node itself) that was unavailable. But often enough the npm registry may be unavailable as well. 

The parent pom of plugins depends on the frontend-maven-plugin (the latter is NOT a jenkins project)

The frontend-maven-plugin downloads specified versions of node so that javascript builds can run with the specified version. 

Unfortunately the parent pom hard codes refrences to both nodejs/dist: 

https://github.com/jenkinsci/plugin-pom/blob/master/pom.xml#L1080

 

And registry.npmjs.org: 

https://github.com/jenkinsci/plugin-pom/blob/master/pom.xml#L1094

 

If INFRA was to offer a mirror of both dist and npm registry (no small ask) then this hard coding could be shifted so builds do not leave the jenkins repo to fetch things they need, vs "external" dependencies. 

 

cc R. Tyler Croy

 

 

---

Originally reported by michaelneale, imported from: Mirror of node dist and npm registry - and usage of it in plugin-pom

• assignee: danielbeck
• status: Resolved
• priority: Minor
• resolution: Fixed
• resolved: 2017-05-18T20:51:06+02:00
• imported: 2022/01/10

[jenkins-infra-bot]

michaelneale:

R. Tyler Croy for your consideration. If you think this should be a dev list discussion, we can move it there. 

 

cc Jesse Glick may be relevant to your interests. Node/npm is so much fun*

[jenkins-infra-bot]

rtyler:

FWIW we're already running an npm repository through artifactory: https://repo.jenkins-ci.org/api/npm/npm

 

Mirroring arbitrary nodejs dist files looks like it would be a bit more tricky. I couldn't find anything with some cursory searching that would allow us to easily mirrors one-off files :/

 

 

[jenkins-infra-bot]

michaelneale:

R. Tyler Croy well - npm registry alone would probably cover the majority of ills. That URL doesnt' return anything for me though. Is it a mirror of registry.npmjs.org ? 

[jenkins-infra-bot]

michaelneale:

Ah I misread the pom - it is getting both binaries for npm and node from specific locations, which is not related to the registry itself, so the hard coding is specific to binaries, not npm artifacts (that is a separate problem which the npm repo could solve). 

so even if we find a place to park these binaries, still doesn't solve the common problem :sadface:

cc Cliff Meyers - if you read the above, we are talking about proxies and mirrors again to reduce jenkins project dependency on npm registry. Any expertise you can add here about setting the registry (if possible) in the parent pom would be appreciated. 

[jenkins-infra-bot]

rtyler:

Navigate to repo.jenkins-ci.org and click "npm" under "Set me up" and that should explain how to use it.

[jenkins-infra-bot]

olblak:

How long does it cachine npm artifacts?

[jenkins-infra-bot]

jglick:

My elevator pitch (for a really tall building) before I found this issue:

In Jenkins 1.x, everything needed to build Jenkins core and plugins could be obtained from our own repo.jenkins-ci.org. If it went down, we had one ticket to raise.

In Jenkins 2, Tom FENNELLY introduced modern JavaScript frameworks. Unfortunately this means that when nodejs.org goes down, as happens from time to time, some of our builds break—core, and any plugins using these frameworks (notably blueocean and workflow-cps). That leads to confusion and loss of developer productivity.

We should be storing whatever node/npm binaries we need, and JS packages we rely on, in Artifactory. jenkinsci/jenkins/war/pom.xml and jenkinsci/plugin-pom/pom.xml would need to be modified accordingly. Probably someone from the Blue Ocean team would be best placed to advise Infra on how to set this up (personally I have only a fuzzy idea of how the current build process works).

[jenkins-infra-bot]

jglick:

I couldn't find anything with some cursory searching that would allow us to easily mirrors one-off files

Well this is trivial in Maven generally: you just pick some groupId / artifactId (not sure if there is a convention for this case), and mvn deploy:deploy-file with suitable arguments. At that point, you can either switch the URL used by download-maven-plugin; or (better) get rid of that plugin altogether and use the standard dependency:copy goal to first make sure the artifacts are cached in the local repository, then copy them to a specified build directory location (since it seems like frontend-maven-plugin expects them to be in a specific directory layout).

That is something any developer with sufficient permissions in repository-permissions-updater could do. About the packages, I am less sure. It seems we are already mirroring them? In principle? According to the Artifactory UI, it would suffice to run

npm config set registry https://repo.jenkins-ci.org/api/npm/npm

but I am not sure what the equivalent would be for frontend-maven-plugin from reading its documentation. But there are some tips here which suggest that everything can be done in maven-frontend-plugin configuration (without a separate download step) after suitable Artifactory setup.

[jenkins-infra-bot]

jglick:

Tom FENNELLY says Cliff Meyers or Yoann Dubreuil or someone experimented with this in the past and had problems with shrinkwrap; unconfirmed.

[jenkins-infra-bot]

cliffmeyers:

Pinging Yoann Dubreuil to validate this comment as I can't recall the outcome of when we rewrote shrinkwrap to use the private registry.

TL;DR:
Based on Jesse Glick's link to the helpful SO post, it seems we ought to consider the following?

• Mirror the versions of npm and node that we know we need in artifactory and configure maven-frontend-plugin appropriately,
• Have our team switch over to using a mirror registry for local development, so that any use of "npm shrinkwrap" will produce a file with URL's that point to our mirror, rather than the public registry. Maybe there's a downside to that I'm not seeing, but we'd have consistency for local vs. CI builds... whereas past attempts would have local devs pulling from public npm registry and CI builds pulling from our mirror (probably bad?)

Details maybe you care about
By default, npm uses "semantic versioning" heavily. If you use "npm install @jenkins-cd/some-package -S", your package.json will be updated to contain a version number like ^1.2.11 for some-package. If you run "npm install" a month from now, maybe you get 1.2.13 installed. Obviously this is bad for build reproducibility.

You can install with an explicit version "npm install @jenkins-cd/some-package@1.2.11 -S -E" which results in a version number of 1.2.11... but semantic versioning will still be used for transitive dependencies. (If "some-package" defines a dep using any of the wildcard version syntaxes, you will have the same build reproducibility issues).

Enter shrinkwrap. It's a command you run that creates a separate npm-shrinkwrap.json file that stores exact version numbers for every dep and transitive dep in the project. You check it into SCM, and future "npm install @jenkins-cd/some-package -S -E" commands update both package.json and npm-shrinkwrap.json. Some points about shrink:

1. It stores fully qualified URL's to the artifacts at the time npm-shrinkwrap.json was created, or when it was updated. This was the source of a problem when we tried to use a mirror for the npm registry: even when configuring npm to use a registry, the URL's stored in shrinkwrap will trump the registry config. Yoann Dubreuil add a sed command to the Jenkinsfile to rewrite the URL's. Unfortunately I can't remember the outcome: it might have caused stability issues, or we may have decided we didn't need the registry mirroring after we upgraded our npm version and found it to be more stable. The other downside to this is approach is that we have different behavior in CI as compared to local builds, unless we get developers to configure the mirror as well (which might actually be the right thing to do).
2. Shrink has been the source of some pain for our team in general, as it can be fickle when you have merge conflicts in the package/shrinkwrap.json files, and also when you're doing local development and want fast feedback without Jenkins. I think myself and Tom FENNELLY have experienced them enough to know the right thing to do, but it's a multi-step process that takes some practice before you get perfect at it. Thorsten Scherler also wrote some nice command line scripts that help automate this process too but they are more specific to Blue Ocean's project layout, not fully generic.

There is a newer package manager from a Google + Facebook collaboration called "yarn" that uses explicit versioning by default and I've read plays nicer with registries. It uses the same package.json format and repo layout as npm, so it's fully compatible, it just creates its own equivalent of npm-shrinkwrap.json (yarn.lockfile) by default so you get explicit versions without any extra work. We haven't evaluated it deeply though, although given all the problems w/ npm to date, I tend to put a lot more faith in a G+F solution.

That said, npm continues to make bug fixes and improvements to shrinkwrap and they plan significant improvements in npm 5 per these release notes for npm 4:
https://github.com/npm/npm/releases/tag/v4.0.0

These releases also include some relevant fixes:
https://github.com/npm/npm/releases/tag/v4.0.1
https://github.com/npm/npm/releases/tag/v4.1.0

[jenkins-infra-bot]

jglick:

Seems like jenkins-infra/jenkins.io is also affected?

[jenkins-infra-bot]

jglick:

Experimenting with whether workflow-cps PR 128 is going to reduce the number of random build failures I see.

[jenkins-infra-bot]

jglick:

We still need to be setting nodeDownloadRoot and yarnDownloadRoot as per here to avoid

[linux-8] [INFO] Downloading https://nodejs.org/dist/v4.0.0/node-v4.0.0-linux-x64.tar.gz to /home/jenkins/.m2/repository/com/github/eirslett/node/4.0.0/node-4.0.0-linux-x64.tar.gz
…
[linux-8] [INFO] Downloading https://github.com/yarnpkg/yarn/releases/download/v0.23.0/yarn-v0.23.0.tar.gz to /home/jenkins/.m2/repository/com/github/eirslett/yarn/0.23.0/yarn-0.23.0./yarn-v0.23.0.tar.gz

[jenkins-infra-bot]

cliffmeyers:

From past discussions, I don't know that either of those packages are actually mirror'd by Artifactory.

[jenkins-infra-bot]

jglick:

See plugin-pom PR 63.

[jenkins-infra-bot]

michaelneale:

Tyler did look at using artifactiry to mirror/cache a directory structure but it was not possible. Jesses approach in the PR could work.

[jenkins-infra-bot]

jglick:

This post claims you can set up Artifactory to mirror the remote repositories. Would need to grab time from R. Tyler Croy to actually set that up, and anyway it is not clear that would automatically interact with the mirror of Artifactory being proposed in INFRA-1176 / INFRA-1180.

[jenkins-infra-bot]

rtyler:

I'm pretty sure I have already done this work as part of INFRA-496, the big problem with Aritfactory and our use of NPM has been the use of "@foo" groups or whatever the hell NPM calls htem

[jenkins-infra-bot]

jglick:

R. Tyler Croy the NPM registry is working fine. I am talking about node/npm/yarn binaries. Not their packages. Perhaps you also set that up, but if so, where is it?

[jenkins-infra-bot]

rtyler:

I did not set that up, sorry I misunderstood the full context while multitasking

[jenkins-infra-bot]

danielbeck:

Jesse Glick Is https://repo.jenkins-ci.org/nodejs-dist/ what you're asking for? It basically mirrors https://nodejs.org/dist/ 

[jenkins-infra-bot]

jglick:

Was this just added? I checked the list of repositories on this server yesterday and I did not see it there.

Yes that be cover npm-based builds. Still need a mirror of the yarn dist site.

[jenkins-infra-bot]

danielbeck:

Jesse Glick Yes, I added it just before I wrote my comment. Tyler asked and I delivered

Is there a real download site for yarn, with directory index, or only GitHub?

[jenkins-infra-bot]

jglick:

So we need to mirror not just http://nodejs.org/dist/ but also http://registry.npmjs.org/npm/-/ and https://github.com/yarnpkg/yarn/releases/download/ according to https://github.com/eirslett/frontend-maven-plugin#installing-node-and-npm (cross-checked in code).

[jenkins-infra-bot]

danielbeck:

I set up https://repo.jenkins-ci.org/npm-tools/ and https://repo.jenkins-ci.org/nodejs-tools/

yarn somehow didn't work (the URLs appear not {{curl}}able either), but since it's GitHub hosted we're not gaining anything from doing that, so I'm not investigating further.

[jenkins-infra-bot]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
.mvn_exec_node
.mvn_exec_yarn
pom.xml
yarn.lock
http://jenkins-ci.org/commit/workflow-cps-plugin/5f871a45f47857e01fc8b80f5ff70447e20b8b1b
Log:
Merge pull request #128 from jglick/yarn

INFRA-1139 Using yarn

Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/457c0ec17b09...5f871a45f478

[jenkins-infra-bot]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
http://jenkins-ci.org/commit/plugin-pom/278ba625d2af0eeea8154086639cc4840e6dfc43
Log:
Merge pull request #63 from jglick/eirslett-cache

INFRA-1139 Load node and npm from the Jenkins Artifactory rather than a third-party server

Compare: https://github.com/jenkinsci/plugin-pom/compare/ef094239a0e5...278ba625d2af