Lack of globally installed node-gyp package causes yarn.lock file to not be created when yarn says it was

[jharris4]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Adding an the uws package to a package.json dependencies section causes yarn to report that it created the lock file successfully, but the lock file does not exist.

If the current behavior is a bug, please provide the steps to reproduce.

Create an empty folder with the following package.json file:

{
  "name": "yarn-lock-bug",
  "version": "1.0.0",
  "description": "repro bug with yarn not creating lock file",
  "author": "jharris4",
  "license": "MIT",
  "dependencies": {
    "engine.io-parser": "~2.1.0"
  },
  "optionalDependencies": {
    "uws": "~0.14.4"
  }
}

Then run yarn install. It will report success Saved lockfile. but the yarn.lock file does not exist.

Removing the dependency and running the same steps (with rm -rf node_modules and rm yarn.lock first if necessary to start from a clean state) does not produce the bug:

{
  "name": "yarn-lock-bug",
  "version": "1.0.0",
  "description": "repro bug with yarn not creating lock file",
  "author": "jharris4",
  "license": "MIT",
  "dependencies": {
    "engine.io-parser": "~2.1.0"
  }
}

Note that running yarn install a second time does correctly create the lock file.

This bug might have something to do with the fact that the uws package uses an install script with node-gyp: https://github.com/uNetworking/bindings/blob/master/nodejs/dist/package.json

What is the expected behavior?

The yarn.lock file should always be created when the yarn cli reports that it was.

Please mention your node.js, yarn and operating system version. MacOS 10.12.5 Node 8.1.2 Yarn 0.24.6

[jharris4]

Actually, just discovered that it seems to be specific to the uws package. Moving from optional dependencies to regular dependencies still causes the lock file to not be created.

[jharris4]

npm install -g node-gyp fixes the issue. It seems that with Node 7, node-gyp was bundled with Node. But with Node 8 it is not...

[domoritz]

Can confirm. This fixes the issue for me.

The fix for this in yarn should be to add this dependency and also actually check whether the lockfile was created instead of just printing a message.

[jharris4]

It appears that yarn is supposed to be adding node-gyp as a dependency, but it's just not working properly: https://github.com/yarnpkg/yarn/blob/23dd84b4ba2e7aff585cadf98b41edad0c925cfe/src/util/execute-lifecycle-script.js#L245

[BYK]

@jharris4 hey, just tried to reproduce this with latest master and using node 8.1.3 and I got a yarn.lock file. Can you try with the latest yarn version that is 0.28.0 or may be from master and let me know if this is still happening?

[jharris4]

@BYK I just checked and the issue is still there with node 8.1.3 and yarn 0.27.5.

I'm using homebrew to install yarn, can you suggest some steps to test out yarn 0.28.x? (it doesn't seem to be published on npm, and the nightlies page doesn't list the tarball either)

[BYK]

@jharris4 - I'm downloading the .tar.gz package from Node's site for Node 8 and then use my local repo for yarn:

git clone git@github.com:yarnpkg/yarn.git
cd yarn
yarn; yarn build;
alias yarn="/path/to/node8 /path/to/yarn/bin/yarn.js"

Then follow your repro steps. May be this is related to Homebrew?

[jharris4]

I used install.sh from the nightlies build, and then copied over the latest tarball contents into ~/.yarn/

Here's the output:

$ yarn --version
0.28.0-20170710.0902
$ ls yarn.lock
ls: yarn.lock: No such file or directory
$ ls node_modules
ls: node_modules: No such file or directory
$ cat package.json
{
  "name": "yarn-lock-bug",
  "version": "1.0.0",
  "description": "repro bug with yarn not creating lock file",
  "author": "jharris4",
  "license": "MIT",
  "dependencies": {
      "engine.io-parser": "~2.1.0",
      "uws": "~8.14.0"
    }
}
$ yarn install
yarn install v0.28.0-20170710.0902
info No lockfile found.
[1/4] 🔍  Resolving packages...
warning engine.io-parser > has-binary2 > isarray@2.0.1: Just use Array.isArray directly
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
⠁ 
⠁ 
⠁ 
⠁ 
info This package requires node-gyp, which is not currently installed. Yarn will attempt to automatically install it. If this f[1/1] ⠁ uws
[1/1] ⠄ uws
[1/1] ⠠ uws
[-/1] ⠠ waiting...
[-/1] ⠠ waiting...
[-/1] ⠠ waiting...
success Saved lockfile.
✨  Done in 2.04s.
$ ls yarn.lock
ls: yarn.lock: No such file or directory

So looks like the bug is still not fixed in 0.28.x :-(

[jharris4]

I tried digging a little further into the issue. I'm a bit of a noob with await/generator syntax, but something seems to be going wrong with the add node-gyp part of the code.

It gets to this line: https://github.com/yarnpkg/yarn/blob/a3ce7c702f644efde783beb8e0b99dc08100f0df/src/cli/commands/_build-sub-commands.js#L33

In the source that line is: const res = await command(config, reporter, flags, args);

But in the distributed/transpiled source it is const res = yield command(config, reporter, flags, args);

I could be mistaken, but that yield looks incorrect to me. In any case, the code following that yield never seems to be executed, and I think that's related to node-gyp not being properly installed...

[BYK]

@jharris4 thanks for digging more and sorry for the late response. Sounds like this may also be a duplicate of #2064 as #3905. What do you think?

[arcanis]

Since #2064 has been closed, this one should be fixed as well.

[jharris4]

@arcanis Is there a version of yarn with the fix that I can install to verify?

[BYK]

@jharris4 https://yarnpkg.com/en/docs/nightly

[BYK]

@jharris4 can we close this now?

[jharris4]

@BYK Sorry for the delay! Unfortunately this bug is still present with yarn 1.0.1.

The workaround of globally installing node-gyp still works, but the yarn lock file is still consistently NOT created if it isn't globally installed.

[BYK]

@jharris4 oh shoot. Are you still on Node 8.1?

[jharris4]

Node 8.4 and Yarn 1.0.1

[kbrgl]

Same issue on Node 8.5 and Yarn 1.1.0. Says it's saved lockfile, even though it hasn't.

[jharris4]

Looks like the issues #4097 and #4653 are related to this issue

[kevinpelgrims]

I can confirm that this is still an issue with Node v9.3.0 and Yarn 1.3.2

warning Error running install script for optional dependency: "/[...]/node_modules/fsevents: Cannot read property 'config' of undefined"
info This module is OPTIONAL, you can safely ignore this error

Installing node-gyp globally (as mentioned in this thread) fixed it for me.

(I think #3905 is a duplicate)

[flmuel]

drove me crazy...

[nloding]

I haven't reviewed all related issues, but yarn also stopped writing to my package.json. Installing node-gyp globally resolved both issues - it now updates the package.json and writes the lockfile.

[victornoel]

Still having problem with node-gyp during yarn install but not always, mainly happening in CI:

[INFO] [4/4] Building fresh packages...
[INFO] info This package requires node-gyp, which is not currently installed. Yarn will attempt to automatically install it. If this fails, you can run "yarn global add node-gyp" to manually install it.
[ERROR] error An unexpected error occurred: "/builds/frontend/node_modules/node-sass: Cannot read property 'getOption' of undefined".
[INFO] info If you think this is a bug, please open a bug report with the information provided in "/usr/local/share/.config/yarn/global/yarn-error.log".
[INFO] info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
[INFO] [1/4] Resolving packages...
[INFO] [2/4] Fetching packages...
[INFO] [3/4] Linking dependencies...
[INFO] [4/4] Building fresh packages...
[INFO] success Installed "node-gyp@3.6.2" with binaries:
[INFO]       - node-gyp

Basically, it feel like there is some kind of concurrency issue: node-gyp is installed because it is needed, and sometimes (not always…) it's not ready yet for installing the dependency that initially required it.

[victornoel]

For the record, only solution I found was to set child_concurrency in the yarn configuration to 1 (see https://yarnpkg.com/en/docs/yarnrc).

[victornoel]

Scratch that, it doesn't work.

[victornoel]

@BYK @arcanis is there some plans to fix those problems once and for all?

[nemonemi]

I am experiencing this even now, 4 years later.

@victornoel, what was your approach to this problem past your last comment here?

[victornoel]

@nemonemi sorry, this was a long time ago and I don't use yarn much lately, but I remember this was related to installing multiple package at the same time, so I would look into concurrency related options (such as child_concurrency mentioned above, maybe it works now?).

[nemonemi]

Cool, thanks for the info. p.s. what do you use now for the client dependencies?

[victornoel]

@nemonemi maven :laughing: j/k, I don't do frontend dev anymore

[nemonemi]

Good switch ;)

[stephannielsen]

6 years later, NPM removes the node-gyp binary from its bundle with v9.7.2 which is released with Node 18.18. Breaking our builds because now yarn tries to install node-gyp automatically and runs every now and then into this issue. Time to move away from yarn v1. :)

https://github.com/npm/cli/pull/6554#issuecomment-1726429283

Leaving a comment primarily for others running into this problem now.