search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.45k stars 2.29k forks source link

Cannot handle yarn v3 and private registry #14756

Open negibokken opened 2 years ago

negibokken commented 2 years ago

How are you running Renovate?

WhiteSource Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

Please select which platform you are using if self-hosting.

No response

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

Hi, thank you for developing such great software!

I met a problem with yarn v3 and private registry (registry.npmjs.org). The problem is that when I specify npmScopes in .yarnrc.yml, I got an error message (Error: Environment variable not found (NPM_TOKEN)) on the lockfile update step in the Docker process (Please see for the detail logs in Relevant debug logs section).

I specified hostRules and encrypted token in renovate.json so Renovate can fetch the version information but Renovate cannot generate lockfile (In this PR, only package.json file is updated).

The minimum repro repo is below.

https://github.com/negibokken/renovate-private-package-repro1

etc.

When I don't specify npmScopes in .yarnrc.yml then Renovate can update lockfile (The error message goes away). But then we can't install it in the local machine because we don't specify ${NPM_TOKEN} for the private registry.

The minimum repro repo for the above case is below. (We can see the lockfile is updated in the PR)

https://github.com/negibokken/renovate-private-package-repro3

So Renovate doesn't require the npmScopes because renovate.json has a private registry setting and Renovate generates npmRegistries from hostRules while processing lockfile.(*1)

Ideas

Currently, I have two ideas to handle this issue.

  1. Pass the environment variables to Docker process
    • We can specify the environment variable here?
  2. Parse .yarnrc.yml and remove npmScopes that includes environment variables (like ${NPM_TOKEN})
    • When renovate overwrites .yarnrc.yml, then remove npmScpes because if we have hostRules in renovate.json, Renovete generates npmRegistries. So we don't need npmScopes for private registry in .yarnrc.yml while Renovate processes the lockfile.(Related to (*1))
    • But we can't handle multiple ${NPM_TOKEN} for a registry. We can specify ${NPM_SCOPE} per registry, not per scope.

(If the task is not too heavy, I can help Renovate team, thanks!)

Relevant debug logs

Logs ``` DEBUG: Fetching Docker image: docker.io/renovate/node(branch="renovate/bokken-npm-test-1.x") DEBUG: Finished fetching Docker image(branch="renovate/bokken-npm-test-1.x") DEBUG: Executing command(branch="renovate/bokken-npm-test-1.x") { "command": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"" } DEBUG: rawExec err(branch="renovate/bokken-npm-test-1.x") { "err": { "killed": false, "code": 1, "signal": null, "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"", "stdout": "\nadded 1 package, and audited 2 packages in 5s\n\nfound 0 vulnerabilities\nUsage Error: Environment variable not found (NPM_TOKEN) in /mnt/renovate/gh/negibokken/renovate-private-package-repro1/.yarnrc.yml (in /mnt/renovate/gh/negibokken/renovate-private-package-repro1/.yarnrc.yml)\n\nYarn Package Manager - 3.2.0\n\n $ yarn \n\nYou can also print more details about any of these commands by calling them with \nthe `-h,--help` flag right after the command name.\n", "stderr": "", "message": "Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"\n", "stack": "Error: Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"\n\n at ChildProcess.exithandler (child_process.js:383:12)\n at ChildProcess.emit (events.js:400:28)\n at ChildProcess.emit (domain.js:475:12)\n at maybeClose (internal/child_process.js:1058:16)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)" } } DEBUG: lock file error(branch="renovate/bokken-npm-test-1.x") { "err": { "killed": false, "code": 1, "signal": null, "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"", "stdout": "\nadded 1 package, and audited 2 packages in 5s\n\nfound 0 vulnerabilities\nUsage Error: Environment variable not found (NPM_TOKEN) in /mnt/renovate/gh/negibokken/renovate-private-package-repro1/.yarnrc.yml (in /mnt/renovate/gh/negibokken/renovate-private-package-repro1/.yarnrc.yml)\n\nYarn Package Manager - 3.2.0\n\n $ yarn \n\nYou can also print more details about any of these commands by calling them with \nthe `-h,--help` flag right after the command name.\n", "stderr": "", "message": "Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"\n", "stack": "Error: Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\":\"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -w \"/mnt/renovate/gh/negibokken/renovate-private-package-repro1\" docker.io/renovate/node bash -l -c \"npm i -g yarn && yarn install --mode=update-lockfile\"\n\n at ChildProcess.exithandler (child_process.js:383:12)\n at ChildProcess.emit (events.js:400:28)\n at ChildProcess.emit (domain.js:475:12)\n at maybeClose (internal/child_process.js:1058:16)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)" }, "type": "yarn" } DEBUG: No updated lock files in branch(branch="renovate/bokken-npm-test-1.x") ```

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description

HonkingGoose commented 2 years ago

Thank you for writing a detailed bug report, and creating a minimal reproduction! ❤️

I found an issue that might be related:

I'll let the maintainers decide what to do with both issues.

negibokken commented 2 years ago

@HonkingGoose Thank you for replying and showing the related issue!

I'll wait for the decision then. Let me know if I can help with Renovate team.

rarkins commented 2 years ago

I think we may want to strip out any sections of the file which use environment variables. IIRC we did this for .npmrc long ago.

rarkins commented 2 years ago

Reproduction forked to https://github.com/renovate-reproductions/14756

Resolution: delete any lines/sections which contain a variable like https://github.com/renovate-reproductions/14756/blob/82428f3c27ddfee20da50894c279c3ef2cdbe745/.yarnrc.yml#L5-L7

After that, any private registry auth would need to be added back, e.g. by hostRules

bobzoller commented 1 year ago

FYI I just worked around this by setting a default value for NPM_TOKEN in .yarnrc.yml. (the other trick was the required trailing slash on the npmRegistries entry.) Here's my final .yarnrc.yml for those interested:

nodeLinker: node-modules
npmRegistryServer: "https://npm.example.com/"
npmRegistries:
  //npm.example.com/:
    npmAlwaysAuth: true
    npmAuthToken: "${NPM_TOKEN:-}"

This plus the hostRules npm entry got renovate unstuck.

viceice commented 1 year ago

I'm working with yarn v3+ and private Gitea registry. I'll post the right config next days

daniluk4000 commented 1 year ago

FYI I just worked around this by setting a default value for NPM_TOKEN in .yarnrc.yml. (the other trick was the required trailing slash on the npmRegistries entry.) Here's my final .yarnrc.yml for those interested:

nodeLinker: node-modules
npmRegistryServer: "https://npm.example.com/"
npmRegistries:
  //npm.example.com/:
    npmAlwaysAuth: true
    npmAuthToken: "${NPM_TOKEN:-}"

This plus the hostRules npm entry got renovate unstuck.

Nice job! Finally we can execute "yarn lint" and other commands without NPM_TOKEN.

jamietanna commented 1 year ago

@bobzoller's note above worked for us 👏, by only adding the trailing slash:

diff --git .yarnrc.yml .yarnrc.yml
index 088d79d..a33a7d2 100644
--- .yarnrc.yml
+++ .yarnrc.yml
@@ -3,6 +3,6 @@ nodeLinker: node-modules
 yarnPath: .yarn/releases/yarn-berry.cjs

 npmRegistries:
-  //registry.yarnpkg.com:
+  //registry.yarnpkg.com/:
     npmAlwaysAuth: true
     npmAuthToken: "${NPM_TOKEN}"