search

renovatebot / app-support

Discussion/support issues for the hosted Renovate App
0 stars 0 forks source link

"Artifacts update problem" with private packages / custom registry (encrypted .npmrc) #31

Closed fabianbru closed 3 years ago

fabianbru commented 5 years ago

What Renovate type are you using? Renovate GitHub App

Describe the bug Private packages or packages from custom registries are not found in the "artifacts" step. So the renovate PR will have the ":warning: Artifact update problem" warning and the failing check "renovate/artifacts".

I have submitted a bug some time ago for issues with those packages (private or custom registry) in the "find new version" step, see renovatebot/renovate#3417. Perhaps this is related?

For what it looks like is that in the "artifacts" step the (encrypted) ".npmrc" content is not used, so no NPM_TOKEN or custom registry?

Notes:

Did you see anything helpful in debug logs? The Dashboard doesn't work at the moment (will update if its working again), but last I looked into there was nothing helpful. Just the warning that is also in the PR comment:

package "@xyz/privatePackage@X.Y.Z" required by "@my/monorepoPackage@X.Y.Z" on the "npm" registry.

To Reproduce See issue renovatebot/renovate#3417. Same setup and same encrypted ".npmrc". After that issue was resolved renovate found new versions, but on my scheduled time the issue above happened.

Expected behavior Artifacts should be build correctly ("yarn.lock") for private packages with NPM_TOKEN or packages from custom registries.

rarkins commented 5 years ago

@fabianbru sorry I missed this issue the other day. I don't understand the full picture so will try to confirm it with you:

  1. You're using the hosted app
  2. Some of your npm packages require an authenticated lookup from a custom repository, others off npmjs.org
  3. You have added an encrypted npmrc file to your renovate config inside package.json
  4. Renovate detects and decrypts the npmrc file fine
  5. Renovate correctly parses and looks up packages from both registries
  6. PRs are raised, but only including package.json and not yarn.lock

Please confirm if the above are correct. Please also try the dashboard again and search for any reference of "npmrc".

When you say it's a yarn monorepo, are you using Yarn Workspaces?

BTW you also mention:

private packages with NPM_TOKEN

Is there any reason you mention NPM_TOKEN? I don't think there's anything to do with the NPM_TOKEN environment variable here, and if so then the above 6 points are incomplete. I thought that the token was embedded in the npmrc field.

fabianbru commented 5 years ago

You summed it up very well, sorry that I relayed on the other issue.

  1. Yes, PRs are only with package.json and the artifact (yarn.lock) fails on generation. My assumption is, that for this step (yarn install ?) the saved npmrc content is missing.

And yes, I am using yarn workspaces.

Is there any reason you mention NPM_TOKEN? I don't think there's anything to do with the NPM_TOKEN environment variable here, and if so then the above 6 points are incomplete. I thought that the token was embedded in the npmrc field.

There is no NPM_TOKEN variable, just the encrpted npmrc to clarify this. Tokens are embedded, yes.

Here are the logs from the dashboard (#59788077): Notes:

rarkins commented 5 years ago

Is there authentication needed for both npmjs private modules as well as a private registry? Are both tokens in the encrypted npmrc?

rarkins commented 5 years ago

And yes I agree that most likely cause is that yarn is not getting all the information it needs from .npmrc

fabianbru commented 5 years ago

Is there authentication needed for both npmjs private modules as well as a private registry? Are both tokens in the encrypted npmrc?

Yes, I need auth tokens for both. One for the official npm registry and another one for the custom registry. My npmrc content looks like (NPM_TOKENS are embedded):

//registry.npmjs.org/:_authToken=ABC-123-NPM_TOKEN1
@fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=ABC-123-NPM_TOKEN2

The content is correct, because I get the PRs with new versions. You fixed an issue related to this in renovatebot/renovate#3417 (the issue I mentioned above).

rarkins commented 5 years ago

Is it pure yarn workspaces or lerna wrapping it?

Next step is to understand:

I suspect we need the npmrc everywhere and right now we’re only writing it to the root

fabianbru commented 5 years ago

We have lerna included, but using it only for starting scripts and managing versions.

Locally we have only one .npmrc file in the project root with similar content as mentioned above, only the tokens are referenced from the environment and are not embedded.

rarkins commented 5 years ago

The logs indicate that the npmrc is being written to the root of the repo and yarn being run from the root too. I am not sure if the logs tell us enough to know if the lookup attempt is from the fontawesome registry or the regular npmjs one. If indeed it’s looking up from the right one then it indicate the authentication (token) is wrong.

fabianbru commented 5 years ago

If indeed it’s looking up from the right one then it indicate the authentication (token) is wrong.

The token is correct, because renovate is creating the PR for this package. If the token is not correct renovate would not be able to get new versions for this package.

Do you have some new insights?

rarkins commented 5 years ago

@fabianbru I want to take another stab at this with fresh eyes. Summarising below.

npm credentials are used in two places:

Your .npmrc credentials would be considered "correct" if Renovate successfully uses them to look up a private module off npmjs.org as well as the font awesome. Ideally we should have observed an example of updates being found for both, however the absence of error logs should also essentially prove it.

Assuming that's the case, we need to answer why Yarn is failing. Are we writing the .npmrc correctly? Do we need to write it in more than one place? Are we somehow corrupting it before writing?

One question I'm not sure if we asked/answered previously: are there any .npmrc files or .yarnrc files checked into the repository?

fabianbru commented 5 years ago

One question I'm not sure if we asked/answered previously: are there any .npmrc files or .yarnrc files checked into the repository?

Yes, there is a .npmrc file checked into the repo. The content is simlar to the encrypted npmrc string, but the npm tokens are not embedded. So our .npmrc file:

//registry.npmjs.org/:_authToken=${NPM_TOKEN}

@fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=${NPM_FONTAWESOME_TOKEN}

In the renovate.json there is also the line "ignoreNpmrcFile": true.

rarkins commented 5 years ago

So in theory we need to replace the .npmrc in your repository with the decrypted one from your config, and everything should work?

It's possible that this is caused by the migration to gitFs, however I thought that particular problem was already handled.

Can you do this for me?

  1. Tick the rebase/retry checkbox on a relevant PR now so that Renovate runs again (and presumably fails again when updating the lock file)
  2. Email the repository name and commit time to support@renovatebot.com so that I can take a deeper look
fabianbru commented 5 years ago

The PR got autoclosed 2 hours ago.

INFO: Failed to look up dependency @fortawesome/angular-fontawesome (@fortawesome/angular-fontawesome)(packageFile="packages/package1/package.json", dependency="@fortawesome/angular-fontawesome")
INFO: Failed to look up dependency @fortawesome/pro-light-svg-icons (@fortawesome/pro-light-svg-icons)(packageFile="packages/package1/package.json", dependency="@fortawesome/pro-light-svg-icons")
INFO: Failed to look up dependency @fortawesome/fontawesome-svg-core (@fortawesome/fontawesome-svg-core)(packageFile="packages/package1/package.json", dependency="@fortawesome/fontawesome-svg-core")

I restored it and checked the renovate rebase checkbox. It run through and found those packages again, but same "Artifact update problem" as we know it.

Edit: And I emailed you the infos earlier today.

rarkins commented 5 years ago

This line confirms that Renovate itself is using the right/valid credentials with the fortawesome npm server:

"dependency":"@fortawesome/pro-light-svg-icons","msg":"1 result(s): 5.8.1"

These pro icons aren't present on registry.npmjs.org so would otherwise error.

Later, the call to yarn reports: error Couldn't find package "@fortawesome/pro-light-svg-icons@5.8.1" required by "@velux/lead-overview@5.3.2" on the "npm" registry.

I'm trying to determine if that's looking up from npmjs or fortawesome. When I create a basic package.json with that dependency included then I see this error instead: error An unexpected error occurred: "https://registry.yarnpkg.com/@fortawesome%2fpro-light-svg-icons: Not found"..

Now, setting a .npmrc like yours but with fake/wrong token for fortawesome, I reproduce almost the same error you got: error Couldn't find package "@fortawesome/pro-light-svg-icons" on the "npm" registry.

My best guess now is that somehow the existing .npmrc is being used rather than the override that's configured. I'll try to reproduce that.

gordlea commented 5 years ago

I am also seeing this error, running self hosted renovate. I've tried it with just an npmrc in the config, and also with a yarnrc, pointing at our private repo.

The funny thing is that it works on my local machine, but not when I run it on jenkins.

gordlea commented 5 years ago

I should note, that on the Jenkins machine, when I run it, there is an .npmrc file that is configured to point at our private registry, and it doesn't work.

gordlea commented 5 years ago

So, I was running renovate via a script in the package.json file of my project, and I was using yarn to run the script on the server. When I changed it to call renovate directly in the Jenkinsfile, it works.

So basically, running renovate via a package.json script started from yarn causes an artifact update error somehow.

@rarkins

rarkins commented 5 years ago

I think these are obviously "related" issues but probably separate problems/solutions to each other. In @gordlea's case it sounds like running the script via yarn causes the "current" .npmrc values to be loaded and passed through and these must somehow be taking precedence.

Geit commented 4 years ago

We're giving Renovate a try on some of our Repos and we've noticed this issue, or a very similar one, appears to be present.

Using the .npmrc we've encrypted for Renovate in the repo locally works fine and is able to fetch packages.

I'll give the self-hosted runner a try with a similar setup, see if I can debug any further and report back.

EDIT:

The self-hosted edition works fine when running on my local machine and correctly regenerates the yarn.lock file. The results of yarn config list are also correct.

I'm not sure how I'd get to something closer to the Hosted App's environment in order to replicate. - Any ideas?

rarkins commented 4 years ago

@Geit maybe I can do some troubleshooting or increased logging from the app side. Perhaps could you create a new/dedicated repo for reproducing this in the app, if you don't have one already? i.e. that way you're not holding up or polluting any real repo with troubleshooting.

Once you've done that, please tell me the repo name or job ID (preferred) of when the problem occurred so that I can take a look.

It sounds like Renovate is failing to write the correct .npmrc file to disk before running yarn.

viceice commented 4 years ago

I've seen such auth problems too on our self-hosted renovate cli (docker-slim).

We have a .npmrc and .yarnrc file in our repo root. I've tried to configure the npmrc renovate config option to add auth, which doesn't work. My workaround is to add a .npmrc with the auth to /home/ubuntu/

As i tried the last time we need to use username and password auth (against proget npm registry), maybe that was a problem too. now we use verdaccio.

Geit commented 4 years ago

Thanks for the prompt reply @rarkins - I've created a small reproduction repo and attached it to the Renovate hosted application. The job ID for its first run was #147251406 and I've also added you as a collaborator to the repo on the off chance that helps.

albanv commented 3 years ago

Hi guys, we encounter the exact same issue using a private registry hosted by us (verdaccio) with the public renovate app:

mbrandau commented 3 years ago
  • then on each renovate PR we encounter the Artifact update problem related to yarn.lock update

I'm experiencing the same issue with verdaccio and a self hosted GitLab instance.

Sy1v4in commented 3 years ago

Hi guys, we encounter the exact same issue using a private registry hosted on Github packages with the public renovate app:

Any update on this problem?

Thanks 🙏

viceice commented 3 years ago

We recommend to use hostrules only instead of npmrc. That way we got most auth issues resolved.

Please open a discussion with detailed reproduction steps. Or search discussions / docs. There are plenty of samples for different registries.

viceice commented 3 years ago

We are also planning to remove npmrc config option at all. So renovate would only read repo npmrc for registry urls.

rarkins commented 3 years ago

I'm going to archive this repository so that we have conversations in one place. Therefore please create an "App Support" discussion here if this problem or suggestion is still relevant: https://github.com/renovatebot/renovate/discussions