Custom npm registry running with jenkins master

[victorb]

The npm registry is still having issues serving packages, returning 404 whenever it feels like it. We should run our own package cache that automatically retries to get packages that gets returned a 404, at least a couple of times.

Would make build times faster but most importantly, make builds more stable.

[victorb]

We've changed to not use the yarn proxy to make sure it's actually the npm registry and we still have failing jobs, so still need to solve this as jobs are randomly failing...

+ ./node_modules/.bin/yarn --registry=https://registry.npmjs.com --mutex network

yarn install v1.7.0

info No lockfile found.

[1/5] Validating package.json...

[2/5] Resolving packages...

error Received malformed response from registry for undefined. The registry may be down.

info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

script returned exit code 1

[victorb]

Idea from @alanshaw is that it could also be that the npm client is more forgiving of errors and/or retries fetches that goes bad, while yarn fails directly. Should test this out.

[victorb]

Tried out with the npm client and still having the same issues. For the record, this also happens to me locally from time to time, with both clients and registries.

[alanshaw]

Any progress? I'm getting this issue a lot today.

[daviddias]

This issue is now wasting countless hours from the dev teams. @VictorBjelkholm can you update us on progress?

[alanshaw]

Clutching at straws here but could we try upgrading yarn to the latest version? 1.7 is quite old now...

[victorb]

I've tried out a couple of proxy registries but they all seem to suffer from the same problem that when the npm registries gives a 404 reply for a package that does exists (which is the issue), it doesn't automatically and transparently retry for the package, so we're still getting random 404s in the pipelines.

We've also already tried latest version of both the npm and yarn cli to no avail.

Currently effort is in implementing a registry proxy that retries X times for package queries that returns 404.

[alanshaw]

🙄 thanks for the update

[victorb]

Temporary measure: implement retry logic for yarn install in JS pipelines, retry it up to 3 times before actually failing should hopefully help a bit at least.

[victorb]

@alanshaw with https://github.com/ipfs/jenkins-libs/pull/31 merged, new builds will now retry yarn install up until 3 times before actually failing.

[victorb]

@alanshaw also, please let me know if you still see builds that are failing

[daviddias]

Use npm on IPFS \o/ @achingbrain

[victorb]

@diasdavid that be great! I wrote the following in #ipfs-dev earlier today:

13:18:50 <@victorbjelkholm> alanshaw achingbrain re https://github.com/ipfs/testing/issues/51, would it make sense to add 404 retry logic to npm-on-ipfs? It'll be hard to test and reproduce as it happens inrequently (locally at least) 13:19:19 <@victorbjelkholm> then we could deploy npm-on-ipfs in the jenkins infra (yay, additional real world testing, but nay for risk in jenkins infra)

So, requires two things:

• Retry logic if there is a 404 answer from the npm registry (could be set via a flag)
• Us being sure that npm-on-ipfs won't introduce more issues than it solves, it's a bit risky

[daviddias]

Us being sure that npm-on-ipfs won't introduce more issues than it solves, it's a bit risky

The risk is mitigated because you can always back up to try regular npm again. Dogfooding our stuff will level our tech dramatically. Always try to use it more, not less :)

[achingbrain]

I think this is a great idea. It would be great to resolve ipfs/infrastructure#432 first though, so it doesn't fall over when no-one's looking at it.

[victorb]

@achingbrain thanks for getting the retry of 404s in place! Only missing piece is having the registry deployed together with jenkins (currently in AWS London I think?) so transfers can happen over private networking and minimize the installation time.