npm prepare script run when lerna is passed --ignore-scripts

[katz12]

What Renovate type are you using?

I am using the hosted app on Github.

Describe the bug

4 days ago I noticed that one of the PRs opened by our renovatebot had encountered an error when running lerna bootstrap. It is failing at a step in our build that runs grunt dist as part of npm run prepare for one of the modules in our monorepo. This step has not changed since we began using renovatebot last August.

I also noticed that checking the box of another dependency in our master issue was not resulting in a PR being opened for quite a while. Eventually a PR was opened but it also had the same error.

Did you see anything helpful in debug logs?

The logs dashboard shows multiple consecutive runs with the label lockfile-error. The relevant portion of those logs are here:

DEBUG: Found compatible npm version(branch="renovate/react-helmet-6.x")
{
  "constraint": "10.15.3",
  "version": "10.15.3"
}
DEBUG: Resolved tag constraint(branch="renovate/react-helmet-6.x")
{
  "image": "renovate/node",
  "tagConstraint": "10.15.3",
  "tagVersioning": "npm",
  "tag": "10.15.3"
}
DEBUG: Fetching Docker image: renovate/node:10.15.3(branch="renovate/react-helmet-6.x")
DEBUG: Finished fetching Docker image(branch="renovate/react-helmet-6.x")
DEBUG: Executing command(branch="renovate/react-helmet-6.x")
{
  "command": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci -- --ignore-scripts --ignore-engines --ignore-platform\""
}
DEBUG: lock file error(branch="renovate/react-helmet-6.x")
{
  "cmd": [
    "yarn install --ignore-scripts --ignore-engines --ignore-platform",
    "lerna bootstrap --no-ci -- --ignore-scripts --ignore-engines --ignore-platform"
  ],
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci -- --ignore-scripts --ignore-engines --ignore-platform\"",
    "stdout": "/home/ubuntu/.npm-global/bin/yarn -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n/home/ubuntu/.npm-global/bin/yarnpkg -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n+ yarn@1.9.4\nadded 1 package in 1.441s\n/home/ubuntu/.npm-global/bin/lerna -> /home/ubuntu/.npm-global/lib/node_modules/lerna/bin/lerna.js\n\n> command-join@2.0.1 postinstall /home/ubuntu/.npm-global/lib/node_modules/lerna/node_modules/command-join\n> npx -p @seangenabe/tnx tnx || exit 0\n\n\n    Thanks for installing command-join!\n\n    If you like this package, be sure to star its repo,\n    and please consider donating:\n\n        https://**redacted**@2.7.1\nadded 311 packages from 170 contributors in 28.733s\nyarn install v1.9.4\n[1/4] Resolving packages...\nDone in 1.27s.\n",
    "stderr": "npx: installed 1 in 2.853s\nlerna info version 2.7.1\nlerna info Bootstrapping 11 packages\nlerna info lifecycle preinstall\nlerna info Installing external dependencies\nlerna info Symlinking packages and binaries\nlerna info lifecycle postinstall\nlerna info lifecycle prepublish\nlerna info lifecycle prepare\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stdout:\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stderr:\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.\n{ Error: Command failed: npm run prepare\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\n    at Promise.all.then.arr (/home/ubuntu/.npm-global/lib/node_modules/lerna/node_modules/execa/index.js:236:11)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n  code: 1,\n  killed: false,\n  stdout:\n   '\\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\\n> grunt dist\\n\\n',\n  stderr:\n   'sh: 1: grunt: not found\\nnpm ERR! file sh\\nnpm ERR! code ELIFECYCLE\\nnpm ERR! errno ENOENT\\nnpm ERR! syscall spawn\\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\\nnpm ERR! spawn ENOENT\\nnpm ERR! \\nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\\n\\nnpm ERR! A complete log of this run can be found in:\\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\\n',\n  failed: true,\n  signal: null,\n  cmd: 'npm run prepare',\n  timedOut: false,\n  pkg:\n   Package {\n     _package:\n      { name: '@datadotworld/dw-bootstrap',\n        description:\n         'The most popular front-end framework for developing responsive, mobile first projects on the web.',\n        version: '3.3.36',\n        keywords: [Array],\n        dwBaseName: 'dw-bootstrap',\n        homepage: 'http://getbootstrap.com',\n        author: 'Twitter, Inc.',\n        scripts: [Object],\n        style: 'dist/css/bootstrap.css',\n        less: 'less/bootstrap.less',\n        main: './dist/js/npm',\n        repository: [Object],\n        bugs: [Object],\n        license: 'MIT',\n        devDependencies: [Object],\n        engines: [Object],\n        files: [Array],\n        jspm: [Object] },\n     _location:\n      '/mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap' },\n  exitCode: 1 }\n",
    "message": "Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci -- --ignore-scripts --ignore-engines --ignore-platform\"\nnpx: installed 1 in 2.853s\nlerna info version 2.7.1\nlerna info Bootstrapping 11 packages\nlerna info lifecycle preinstall\nlerna info Installing external dependencies\nlerna info Symlinking packages and binaries\nlerna info lifecycle postinstall\nlerna info lifecycle prepublish\nlerna info lifecycle prepare\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stdout:\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stderr:\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.\n{ Error: Command failed: npm run prepare\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\n    at Promise.all.then.arr (/home/ubuntu/.npm-global/lib/node_modules/lerna/node_modules/execa/index.js:236:11)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n  code: 1,\n  killed: false,\n  stdout:\n   '\\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\\n> grunt dist\\n\\n',\n  stderr:\n   'sh: 1: grunt: not found\\nnpm ERR! file sh\\nnpm ERR! code ELIFECYCLE\\nnpm ERR! errno ENOENT\\nnpm ERR! syscall spawn\\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\\nnpm ERR! spawn ENOENT\\nnpm ERR! \\nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\\n\\nnpm ERR! A complete log of this run can be found in:\\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\\n',\n  failed: true,\n  signal: null,\n  cmd: 'npm run prepare',\n  timedOut: false,\n  pkg:\n   Package {\n     _package:\n      { name: '@datadotworld/dw-bootstrap',\n        description:\n         'The most popular front-end framework for developing responsive, mobile first projects on the web.',\n        version: '3.3.36',\n        keywords: [Array],\n        dwBaseName: 'dw-bootstrap',\n        homepage: 'http://getbootstrap.com',\n        author: 'Twitter, Inc.',\n        scripts: [Object],\n        style: 'dist/css/bootstrap.css',\n        less: 'less/bootstrap.less',\n        main: './dist/js/npm',\n        repository: [Object],\n        bugs: [Object],\n        license: 'MIT',\n        devDependencies: [Object],\n        engines: [Object],\n        files: [Array],\n        jspm: [Object] },\n     _location:\n      '/mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap' },\n  exitCode: 1 }\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci -- --ignore-scripts --ignore-engines --ignore-platform\"\nnpx: installed 1 in 2.853s\nlerna info version 2.7.1\nlerna info Bootstrapping 11 packages\nlerna info lifecycle preinstall\nlerna info Installing external dependencies\nlerna info Symlinking packages and binaries\nlerna info lifecycle postinstall\nlerna info lifecycle prepublish\nlerna info lifecycle prepare\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stdout:\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stderr:\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.\n{ Error: Command failed: npm run prepare\nsh: 1: grunt: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\n\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\n> grunt dist\n\n\n    at Promise.all.then.arr (/home/ubuntu/.npm-global/lib/node_modules/lerna/node_modules/execa/index.js:236:11)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n  code: 1,\n  killed: false,\n  stdout:\n   '\\n> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap\\n> grunt dist\\n\\n',\n  stderr:\n   'sh: 1: grunt: not found\\nnpm ERR! file sh\\nnpm ERR! code ELIFECYCLE\\nnpm ERR! errno ENOENT\\nnpm ERR! syscall spawn\\nnpm ERR! @datadotworld/dw-bootstrap@3.3.36 prepare: `grunt dist`\\nnpm ERR! spawn ENOENT\\nnpm ERR! \\nnpm ERR! Failed at the @datadotworld/dw-bootstrap@3.3.36 prepare script.\\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\\n\\nnpm ERR! A complete log of this run can be found in:\\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-06-08T21_10_03_599Z-debug.log\\n',\n  failed: true,\n  signal: null,\n  cmd: 'npm run prepare',\n  timedOut: false,\n  pkg:\n   Package {\n     _package:\n      { name: '@datadotworld/dw-bootstrap',\n        description:\n         'The most popular front-end framework for developing responsive, mobile first projects on the web.',\n        version: '3.3.36',\n        keywords: [Array],\n        dwBaseName: 'dw-bootstrap',\n        homepage: 'http://getbootstrap.com',\n        author: 'Twitter, Inc.',\n        scripts: [Object],\n        style: 'dist/css/bootstrap.css',\n        less: 'less/bootstrap.less',\n        main: './dist/js/npm',\n        repository: [Object],\n        bugs: [Object],\n        license: 'MIT',\n        devDependencies: [Object],\n        engines: [Object],\n        files: [Array],\n        jspm: [Object] },\n     _location:\n      '/mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap' },\n  exitCode: 1 }\n\n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:311:20)\n    at ChildProcess.EventEmitter.emit (domain.js:482:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
  },
  "type": "lerna",
  "lernaClient": "yarn"
}
DEBUG: No updated lock files in branch(branch="renovate/react-helmet-6.x")
DEBUG: Branch timestamp: 2020-06-08T04:59:07.232Z(branch="renovate/react-helmet-6.x")
DEBUG: PR is less than a day old - raise error instead of PR(branch="renovate/react-helmet-6.x")
DEBUG: Passing lockfile-error up(branch="renovate/react-helmet-6.x")
INFO: Lock file error - aborting

To Reproduce

I do not own our renovatebot configuration, so I am not sure exactly how to set up a similar repository.

I have searched elsewhere for similar issues but have not found any.

Additional context

I looked around at various renovate docker repos (node, npm, buildpack) but have not found any recent changes that seem like they would cause grunt to not be available.

[rarkins]

It's not our intention to be running grunt in order to just update lockfiles with lerna. If you are able to reproduce this problem in a public repo then we can take a look at whether it's possible to avoid it.

[viceice]

@rarkins looks like --ignore-scripts is somehow ignored

[rarkins]

I'm wondering if lerna explicitly calls prepare or if --ignore-scripts is not being passed to yarn install.

Also wondering if we can stop using lerna bootstrap when Yarn Workspaces are in use and instead just call yarn install directly. I had thought calling lerna bootstrap was more future-compatible but happy to skip it if it's causing problems.

[mpeyper]

Could this be the same issue that's causing the dependency updates in my libraries to fail?

https://github.com/ioof-holdings/redux-subspace/pulls https://github.com/ioof-holdings/redux-dynostore/pulls

They are all failing with a comment like this where it's failing to find a dependency used in a prepublish script. These bootstrap commands work when I run them locally, and I did not have issues with the renovate updates until a few days ago, so not sure what may have changed to cause it.

[rarkins]

We were probably doing a full install (unintentionally) of dependencies until recently.

[mpeyper]

Sorry @rarkins, I don't quite follow.

Should the prepublish script not be running? Do I need to change anything in my repo and/or renovate config to get them passing again?

[rarkins]

Correct - the prepublish script shouldn’t be running. If it was running before, it was accidental. But now, we need to get it all working again one way or another. First task is to try to work out why Lerna isn’t honouring our —ignore-scripts setting.

[mpeyper]

Thanks for the clarification.

I notice there was recently an issue that around this that has been closed (#6433) and an older issue in the lerna repo about it being ignored with yarn (lerna/lerna#1800). Not sure if either are relevant here.

[katz12]

Thanks yall for engaging with this with little repro information.

It would make a lot of sense if this prepare step just wasn't being called before but now is. The issue with grunt having nothing to do with renovatebot, only my configuration.

It is also interesting that both yarn and lerna are used. Locally I always use lerna bootstrap and similar to @mpeyper it works when run locally. We have played around with the idea of switching to yarn workspaces in the past, so there may be something about our setup that makes renovatebot think it should use yarn as well.

Although based on the error output, the issue seems to be happening during the lerna bootstrap step. Here is an example with better whitespace, it looks a lot like the error on @mpeyper's pulls:

npx: installed 1 in 2.901s
lerna info version 2.7.1
lerna info Bootstrapping 11 packages
lerna info lifecycle preinstall
lerna info Installing external dependencies
lerna info Symlinking packages and binaries
lerna info lifecycle postinstall
lerna info lifecycle prepublish
lerna info lifecycle prepare
lerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'
lerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stdout:

> @datadotworld/dw-bootstrap@3.3.36 prepare /mnt/renovate/gh/datadotworld/ui/css-components/dw-bootstrap
> grunt dist

lerna ERR! npm run prepare [@datadotworld/dw-bootstrap] Output from stderr:
sh: 1: grunt: not found

[rarkins]

There shouldn't be any npx call anymore. Is that log extract a little old?

It is also interesting that both yarn and lerna are used. Locally I always use lerna bootstrap and similar to @mpeyper it works when run locally.

The first yarn was intended for updating the "root" lock file while lerna bootstrap was for the child packages. We can reconsider this if it's unnecessary. Are you using a single lock file or many?

[katz12]

We are using multiple lock files. And I believe it is necessary to use yarn to update the root lock file. Again, sorry, I do not own our configuration so I don't understand it fully.

That log extract was created on June 5 around 10AM UTC and posted by renovatebot. Another one was posted to a different renovatebot PR 16 hours ago and it also includes the line about npx.

Edit: @rarkins I just had renovatebot create a new PR this morning (June 10). It hit the same issue and still has npx: installed 1 in 2.698s in the output.

[mpeyper]

I'm actually seeing a mixture of npx and regular lerna output in "Artifact update problem" comments:

https://github.com/ioof-holdings/redux-dynostore/pull/287#issuecomment-638979371 https://github.com/ioof-holdings/redux-dynostore/pull/286#issuecomment-638503438

Because some of our dependencies have a restriction of using a singleton reference (react, react-redux), we use the --hoist option for lerna bootstrap, which hoists as many dependencies it can (only leaves the ones it can't dedupe) to the root of the project so that they are shared between the packages in the monorepo. The result of this is that we have a single lock file in the root only that has all the dependencies for the root project and all the sub-packages combined.

Also, for clarity, I'm not using yarn at all in my project, just regular old npm and lerna.

[katz12]

Based off the problem and discussion in https://github.com/renovatebot/renovate/issues/6433 I wonder if https://github.com/renovatebot/renovate/pull/6434 did not have the desired effect.

[rarkins]

It seems removing npx didn’t resolve the problem, although was still the right thing to do.

[mpeyper]

Any update on this? Is there anything I can do to help it along? I'm unfamiliar with the codebase, but willing to learn.

[rarkins]

The best thing would be to confirm if lerna is at fault here. e.g. why is it running npm prepare scripts if we pass --ignore-scripts.

[mpeyper]

So my investigations have led me to this:

Calling lerna as you are now results in the following command being run (with some variability in the complete list of option that get passed)

lerna bootstrap --no-ci -- --ignore-scripts --no-audit

If I run lerna in my project this way it results in the scripts not being ignored. However, the scripts are ignored if I change the command to the following

lerna bootstrap --no-ci --ignore-scripts -- --no-audit

The difference here is that the --ignore-scripts is being passed as a lerna option instead of being passed onto the underlying package manager.

Now this is where things get a bit funky. If the repo is configured to use yarn workspaces, but still uses lerna for monorepo management, the bootstrap command defers to yarn install and according to this lerna issue, the original version is the correct way to ignore scripts. This is the last reference I have found to have the option is handled between lerna and yarn, but I have not tested whether or not this is still true.

I ran a quick test and running the command like this

lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --no-audit

This worked with NPM and lerna and I'm assuming it would also work for the yarn case described above, but it does look a bit odd and I'm not sure how correct it actually is.

Let me know what you think and if you want I can put together a PR to move the option to the other side of the -- and/or include the additional one. Also, please let me know if you've got any ideas or if there is another course of action you'd prefer to take.

[viceice]

The question is, do we need to run Lerna to update the lockfiles or would it be enough to run the package manager only.

[mpeyper]

For what it's worth, I just reconfigured one of my project to use yarn, and then to use yarn with workspaces, and both cases are ignoring the scripts when just using

lerna bootstrap --no-ci --ignore-scripts -- --no-audit

So moving it to the other side of the -- should work in all three cases (I don't believe lerna supports any others package managers yet).

The question is, do we need to run Lerna to update the lockfiles or would it be enough to run the package manager only.

I think you want lerna to manage the lock files. The managed packages are not included in the lock files and using options like hoist can result in significantly different lock files than if you used the package managers directly in each package. I could be wrong though and just not understand how renovate actually works well enough.

[rarkins]

Last time I looked it seemed like lerna + yarn workspaces works ok if you just call yarn directly, however my concern was that we can't be sure that lerna won't change something in future, so lerna bootstrap is most safe.

[pharindoko]

Hey,

I do get these errors as well. I can confirm that I still have the same issue with lerna bootstrap

npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that. lerna notice cli v3.22.1 lerna info Bootstrapping 3 packages lerna info Installing external dependencies lerna info Symlinking packages and binaries lerna info lifecycle json-serverless-lib@1.5.42~prepare: json-serverless-lib@1.5.42 sh: 1: tsc: not found npm ERR! code ELIFECYCLE npm ERR! syscall spawn npm ERR! file sh npm ERR! errno ENOENT npm ERR! json-serverless-lib@1.5.42 compile: tsc -p . npm ERR! spawn ENOENT npm ERR! npm ERR! Failed at the json-serverless-lib@1.5.42 compile script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in: npm ERR! /tmp/renovate-cache/others/npm/_logs/2020-06-19T14_56_17_256Z-debug.log lerna info lifecycle json-serverless-lib@1.5.42~prepare: Failed to exec prepare script lerna ERR! lifecycle "prepare" errored in "json-serverless-lib", exiting 1

here`s my script config:

  "scripts": {
    "test": "npx jest",
    "debug": "nodemon --watch 'src/**/*.ts' --ignore 'src/**/*.spec.ts' --exec node --inspect-brk -r ts-node/register example/index.ts",
    "start": "nodemon --watch 'src/**/*.ts' --ignore 'src/**/*.spec.ts' --exec 'ts-node' example/index.ts",
    "type-check": "tsc --noEmit",
    "type-check:watch": "npm run type-check -- --watch",
    "build": "tsc -b",
    "prepare": "npm run compile",
    "pretest": "npm run compile",
    "posttest": "npm run check",
    "check": "gts check",
    "clean": "gts clean",
    "compile": "tsc -p .",
    "fix": "gts fix"
  }, 

[pharindoko]

@rarkins @mpeyper Is there any update on this. Using npm at least I do get this error. Interesting is that this worked minimum for a year and stopped a few weeks ago.

[mpeyper]

Sorry, I have not moved any further with it.

I was going to put together a PR for consideration, but I've been a bit slammed at work (EOFY in Australia and I working in financial services).

[pharindoko]

I will test it locally when I have time but cannot promise it as well....

[renovate-release]

:tada: This issue has been resolved in version 21.16.1 :tada:

The release is available on:

• GitHub release
• 21.16.1

Your semantic-release bot :package::rocket:

[katz12]

It doesn't look like that did the trick for us:

INFO: Repository started
{
  "renovateVersion": "21.16.1"
}

...

(branch="renovate/react-overlays-3.x")
{
  "command": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\""
}
DEBUG: lock file error(branch="renovate/react-overlays-3.x")
{
  "cmd": [
    "yarn install --ignore-scripts --ignore-engines --ignore-platform",
    "lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform"
  ],
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/datadotworld/ui\":\"/mnt/renovate/gh/datadotworld/ui\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/datadotworld/ui\" renovate/node:10.15.3 bash -l -c \"npm i -g yarn@1.9.4 && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g lerna@2.7.1 && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\"",
    "stdout": "/home/ubuntu/.npm-global/bin/yarn -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n/home/ubuntu/.npm-global/bin/yarnpkg -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n+ yarn@1.9.4\nadded 1 package in 1.206s\n/home/ubuntu/.npm-global/bin/lerna -> /home/ubuntu/.npm-global/lib/node_modules/lerna/bin/lerna.js\n\n> command-join@2.0.1 postinstall /home/ubuntu/.npm-global/lib/node_modules/lerna/node_modules/command-join\n> npx -p @seangenabe/tnx tnx || exit 0\n\n\n    Thanks for installing command-join!\n\n    If you like this package, be sure to star its repo,\n    and please consider donating:\n\n        https://**redacted**@2.7.1\nadded 310 packages from 169 contributors in 27.26s\nyarn install v1.9.4\n[1/4] Resolving packages...\nDone in 1.25s.\n",
    "stderr": "npx: installed 1 in 2.769s\nlerna info version 2.7.1\nlerna info Bootstrapping 11 packages\nlerna info lifecycle preinstall\nlerna info Installing external dependencies\nlerna info Symlinking packages and binaries\nlerna info lifecycle postinstall\nlerna info lifecycle prepublish\nlerna info lifecycle prepare\nlerna ERR! execute Error occured with '@datadotworld/dw-bootstrap' while running 'npm run prepare'\nlerna ERR! npm run prepare [@datadotworld/dw-bootstrap]
...

[viceice]

looks like lerna is calling the prepare script explicitly

[viceice]

[rarkins]

Is it ok if we let that step fail?

[viceice]

🤔 --ignore-scripts should do ignore lifecycle scripts since 2.9.0 so to get older versions working, we need to full install packages or patch lerna to not run scripts

[rarkins]

@viceice I think in that case it's not worth patching lerna or writing extra code for people running 2+ year old releases. @katz12 can you upgrade and test again?

[mpeyper]

It looks like the new version is working for my setup.

A few of the Travis checks on the PRs are complaining that the package-lock.json doesn't match the package.json, but others worked fine, so I'm don't think renovate is causing the issue there.

[katz12]

Sorry for the delay but I can confirm that upgrading to ^2.9.0 worked for me too. Thanks for the help everyone!

[viceice]

@katz12 Thanks for verify

[trusktr]

Can we run prepare scripts only for dependencies of our lerna-managed packages?

i.e. I'd like to skip prepare for all the packages in my repo that are managed by Lerna, but I'd still like for prepare to run for any dependencies of my packages which are not managed by lerna (those ones are never linked).

[viceice]

@trusktr please open a new issue with your requirement.