yarn install fails with `ENOENT: no such file or directory` occasionally

[NiGhTTraX]

Running yarn install as part of a build step for a Docker image based on node:7 fails on Travis CI with ENOTEMPTY, EEXISTS errors. It always seems to error on the webdriverio package.

yarn install v0.19.1
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/webdriverio/-/webdriverio-4.6.2.tgz: ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol/timeouts.js'".

When Travis runs yarn install as part of the install phase it works just fine. The error only happens when building a Docker image.

Repo which reproduces this issue.

node:7 OS: Docker + Travis CI yarn: 0.19.1 package.json yarn.lock

I've tried installing yarn both with npm install -g and with apt and both methods cause the failure on Travis.

Weirdly enough, the image builds successfully on my local machine which runs Ubuntu 16.04.1 LTS with Docker version 1.13.0, build 49bf474.

[Redmega]

This is causing my Travis builds to fail almost every single time, with a few different packages. Is there a solution yet? error An unexpected error occurred: "https://registry.yarnpkg.com/apollo-client/-/apollo-client-1.8.0.tgz: ENOENT: no such file or directory, utime '/var/lib/jenkins/.cache/yarn/v1/npm-apollo-client-1.8.0-3b5d1976a06a0f82b2fc66fe71754868193dadb9/flow-typed/npm/webpack_vx.x.x.js'".

[benmerckx]

@Redmega Same here, but this works:

yarn install --network-concurrency 1

[victornoel]

Which version are you using? This is meant to be fixed already...

Le 8 août 2017 6:37 PM, "Ben Merckx" notifications@github.com a écrit :

@Redmega https://github.com/redmega Same here, but this works:

yarn install --network-concurrency 1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yarnpkg/yarn/issues/2629#issuecomment-321011749, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ0z5qFb7gSW4w14_RbFNjsn4sRYV78ks5sWI7hgaJpZM4L3JbX .

[Redmega]

@victornoel I am using v0.27.5 on the jenkins machine, same as my local.

[BYK]

Please try the nightlies: https://yarnpkg.com/en/docs/nightly

[normansander]

Removing the yarn.lock file and yarn install fixed the problem for me.

[ajcrites]

This also causes my Jenkins builds to fail occasionally. Usually it works after a second try but will fail again later.

[victornoel]

@ajcrites @Redmega @headione @benmerckx you should open another issue if you are experiencing this kind of problem. This issue has been fixed for sure, so your problem must be a different one, even if it shows some similar symptoms. I'm pretty sure there is more chance for your problem to be solved if you open another issue :)

[floric]

We have the same problem, doing parallel builds of packages in Jenkins with Node 8.5. We currently need to stick to 0.27.5, until 1.0.2 is released fixing another bug. But thanks for your support and work anyway :)

[Niceplace]

@floric I'm getting the same issue with the same context (Jenkins + Parallel) with node 8.9.4, has your issue been resolved ?

Edit: I'll try to use 8.11.1 to see if it includes a latest version of yarn without the bug.

[BYK]

@Niceplace you may wanna try the --mutex option: https://yarnpkg.com/en/docs/cli#toc-concurrency-and-mutex

We have plans for adding better per-package locking to avoid this soon.

[FezVrasta]

I'm having intermittent errors with both ENOENT: no such file or directory, chmod and ENOENT: no such file or directory, lstat trying to run yarn --mutex=network on the root of a monorepo with Yarn workspace enabled...

It doesn't seem to be consistent, I get either one or the other randomly. (1.6.0 and node 8.11.1 and 9.11.1)

Specifically, the errors are:

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/Users/federicozivolo/test/packages/foobar/node_modules/detect-port-alt'".

and

error An unexpected error occurred: "ENOENT: no such file or directory, chmod '/Users/federicozivolo/test/packages/foobar/node_modules/jest/node_modules/.bin/jest'".

[nieltg]

I'm running Yarn 1.7.0 and I'm having the similar error. Yarn finally manages to install the package after several runs.

An unexpected error occurred: "ENOENT: no such file or directory, lstat '/home/nieltg/.cache/yarn/v1/npm-npm-registry-client-8.5.1-8115809c0a4b40938b8a109b8ea74d26c6f5d7f1/lib/dist-tags/fetch.js'".

EDIT: I've used yarn --network-concurrency 1 but the error still occurs on me. Here is another sample of the error and yarn-error.log file.

An unexpected error occurred: "ENOENT: no such file or directory, copyfile '/home/nieltg/.cache/yarn/v1/npm-core-js-2.5.7-f972608ff0cead68b841a16a932d0b183791814e/library/fn/date/now.js' -> '/mnt/c/Users/nieltg/Projects/React/React-16-Demo/node_modules/core-js/library/fn/date/now.js'".

[rdsedmundo]

I'm using Yarn 1.7.0. And I can confirm the same behavior still happening to me.

It's completely random. Sometimes it happens, sometimes it doesn't.

The last I received was:

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/root/.yarn-cache/v1/npm-@storybook/addon-actions-3.4.5-ba0d0c0c74357c0852e0b890b40

[ghost]

I'm seeing this error quite frequently with Yarn 1.9.2 on Windows Subsystem for Linux.

[antonovicha]

Today we got similar problems with broken packages on Jenkins CI where pipeline run yarn install in parallel. It was working fine jest few days ago. Fixed with yarn install --network-concurrency 1 (as mentioned in a comment). Performance did not degraded much: ~7 sec -> ~8 sec.

[tommedema]

Why was this closed? It still happens:

Toms-MacBook-Pro-2:design-to-code tommedema$ yarn install
yarn install v1.9.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/Users/tommedema/projects/vg/design-to-code/packages/vgcli/node_modules/fs-extra'".
info If you think this is a bug, please open a bug report with the information provided in "/Users/tommedema/projects/vg/design-to-code/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Toms-MacBook-Pro-2:design-to-code tommedema$ yarn install --network-concurrency 1
yarn install v1.9.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 24.85s.

Note that in my case it disappeared after doing yarn remove fs-extra and yarn add fs-extra in two packages, effectively upgrading this dependency.

[nieltg]

Hi, I think I found something.

I was dabbling with a piece of code which list files in a specified directory recursively using fs and rxjs and I discovered that it was likely to fail if I don't wait for lstat call to be finished before calling another lstat.

I've made a simple NPM package, async-dirtree-test, to test if your environment is affected or not with this. I'm using WSL and I found that it is likely to fail while handling a directory which has many child directories, like node_modules, even with a low number of concurrency.

Well, I still don't know if this problem is WSL specific or not. Right now I can't test it in another environment, like Linux, Mac, etc.

[rainabba]

@nieltg I wanted to share an observation that might help shape some of the others. I'm using Docker CE in WSL and Docker for Windows as my host so when I work with Docker in WSL, it feels native, but in reality the host operates in the Windows world natively (so my Dockerfiles actually resolve /c/foobar to c:/foobar in the Docker engine). This is incredibly relevant when I use bindings (inside my container, I'm mounting my local folders so that /usr/src inside the container is ultimately at c:/src/foobar (though my Dockerfile would show the binding as /c/src/foobar:/usr/src (see the automatic translation in the path?)

This distinction is important because if I yarn install inside one of those local folders, inside my container, I get the same errors I do directly in WSL (no Docker involved).

On the other hand.... if I just mkdir /tmp/src && cp ./package.json /tmp/src/ && cd /tmp/src && yarn install, everything succeeds perfectly and I can just mv /tmp/src/node_modules /c/src/foobar/ and I'm good, so that's my present workaround. Mind you that /tmp exists as a docker store (all the IO looks like a single file to the OS because it's effectively a partition in a file).

I know that involving docker isn't ideal here, but it seems to suggest that rapid file handles might be an issue since IO itself isn't and might help others work around this.

...got distracted and submitted too soon. Anyhow, my brain is elsewhere at the moment, but I'll come back later and see if I can devise a test using your approach and Docker to draw further conclusions.

[konradjurk]

REOPEN

[Marclev78]

We have recently started seeing the same sort of error using yarn 1.10.1, running a CI build in Azure Devops (formerly Visual Studio Team Services).

The actual dependency that is failing seems to be random best I can tell, but yarn install is falling over intermittently with the ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn........ error. One time it the build will work, the next it will fail.

The workaround of yarn install --network-concurrency 1 seems to work for us.

[malware27]

@Marclev78 same error but yarn install --network-concurrency 1 doesn't seem to work for me

[Fabianopb]

@Marclev78 same thing here, using yarn 1.10.1 in Azure Devops and getting the error:

Error: https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: ENOENT: no such file or directory, utime 'C:\Users\grpsshagent\AppData\Local\Yarn\Cache\v1\npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636\fn\string\pad-left.js'

Locally everything works as expected.

[jsatk]

I'm here to simply say I too am seeing this error.

error An unexpected error occurred: "ENOENT: no such file or directory, chmod '/usr/local/opt/asdf/installs/nodejs/8.12.0/.npm/bin/atob'".

Unfortunately I think I have to abandoning yarn for my global node binaries and moving back to npm until this is fixed.

[Nopik]

Sadly, this problem started to plague our CI builds recently, too ;(

[error-bruno]

@rainabba suggestion worked for me on WSL

[rainabba]

I think that write and read operations behave differently also. Even with my hacks, I frequently see errors while calling node's fs.writeFile (wrapped with bluebird promisify though). In every single instance, I can confirm the file exists immediately after I get the write error.

I'm sending a string (XML file content) to fs.writeFile(), which ultimately calls the following, but I'm not sure if I'm prepared to undertake the challenge that be required to setup a custom build with additional debugging output from this C++ project, so that I can confirm exactly where the right is actually feeling according to node or this C++ module.

Bottom line is that the writes are not failing, but node believes they are so the scenario that makes sense to me is that the c+ plus module is succeeding, but internally it checks for the file and fails, then reports not feel your back to node and then the actual write occurs so that when I go to check for the file, it is there and the error makes no sense.

https://github.com/nodejs/node/blob/master/src/node_file.cc#L1795

[victornoel]

@bestander any way to get that issue reopened? Clearly this is not fixed and impacting a lot of people.

[arichiardi]

Confirming this still happens with yarn 1.12 and Azure Pipelines.

[bestander]

Thanks for confirming everyone. Looks like there are multiple reasons for that error. I'll reopen the issue, community help to debug this is needed though.

[TTouka]

also happens with yarn 1.11, but not with 1.10

[jhalborg]

@bestander - Related? https://github.com/yarnpkg/yarn/issues/6312

If so, there's some fine repro work inthere

[limonte]

I'm affected by this issue as well.

Windows 10 / WSL

"ENOENT: no such file or directory, lstat '/mnt/c/Users/<username>/.cache/yarn/v4/<random_file_in_random_package>"

[rafsawicki]

@limonte WSL had an error for a while that it would randomly throw a similar error when executing npm install / yarn install. It was happening when a lot of files at once were copied to the hard drive. Please make sure that you're on the latest Windows version (1809 or higher), as it might not be caused by the yarn itself.

[mobilutz]

We are also seeing the problem of Extracting tar content of undefined.

error https://registry.yarnpkg.com/eslint/-/eslint-4.19.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/tmp/yarncache.KTKNZ/v4/npm-eslint-4.19.1-32d1d653e1d90408854bfb296f076ec7e186a300/node_modules/eslint/lib/rules/no-compare-neg-zero.js'"

So far we mitigated this with using only one concurrent network connections with the option --network-concurrency 1. But this is more a temporary solution.

[krns]

I also can confirm the issue on node:11.5.0-alpine.

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/app/node_modules/<random_pacakge>

[cinderblock]

I noticed that the problems seemed to be related to linking to the git repository version of packages.

Reproduce

package.json

{
  "dependencies": {
    "react-navigation-core": "https://github.com/react-navigation/react-navigation-core",
    "react-navigation-hooks": "https://github.com/react-navigation/react-navigation-hooks"
  }
}

rm -rf node_modules && yarn cache clean && yarn

Workaround

Setting network-concurrency 1 fixes the problem for me every time.

Running npm install also works.

Notes

Removing either package from the dependency list doesn't cause the error, nor does using the published npm versions of those packages.

It seems to throw a different error every time. It seems to happen randomly to different files and with slightly different errors.

error https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod  '/home/cameron/.cache/yarn/v4/npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636/node_modules/core-js/library/modules/es6.reflect.apply.js'"

Variations

• ENOENT: no such file or directory, chmod
• ENOENT: no such file or directory, stat
• ENOENT: no such file or directory, open
• EEXIST: file already exists, mkdir

Other messages

info There appears to be trouble with your network connection. Retrying...

[Nopik]

Indeed, I've just tried to reproduce using your package.json and error showed up on first try.

[rainabba]

What layer does the WSL filesystem interact with the NTFS fs that's already in place?

Are you folks seeing this error in a mounted drive (/c or /mnt/c for common examples), or outside one of those mounts? Mind testing aan alternate (~/. for example) and reporting any difference?

My intuition is nagging at me, but I may be confusing issues with my docker experiences and I need to check this independently.

[freemanercom]

[2/4] Fetching packages... error https://registry.yarnpkg.com/smartwrap/-/smartwrap-1.0.10.tgz: Extracting tar content of undefined failed, the file appears to be co rrupt: "ENOENT: no such file or directory, open 'C:\Users\Administrator\AppData\Local\Yarn\Cache\v4\npm-smartwrap-1.0.10-873ef350d 4ee1262fed4a80a55634d86ae1faf48\node_modules\smartwrap\ejq'" info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.

[w-biggs]

Are you folks seeing this error in a mounted drive (/c or /mnt/c for common examples), or outside one of those mounts? Mind testing aan alternate (~/. for example) and reporting any difference?

Is there a consistently reproducible case where this is happening? It's been pretty random for me, but I do all of my yarn add-ing on a mounted drive and this occurs frequently.

I was able to reproduce https://github.com/yarnpkg/yarn/issues/2629#issuecomment-451638917 in both a mounted drive and in ~.

I also tried to reproduce https://github.com/yarnpkg/yarn/issues/2629#issuecomment-282745896 but it kept failing to fetching the last package, which I'm pretty sure is unrelated.

[lizozom]

I had the same issue in the last few hours. yarn was randomly failing to install various packages, showing the errors that were mentioned above.

I tried reseting yarn cache and re-installing and running with network-concurrency 1, neither of which had worked.

What solved the problem for me, was switching to a different network (just used my phone AP instead of WiFi) and everything worked like magic.

I have a hunch that this issue might be related to mishandled recovery for some very specific network errors. Will look into it later.

[mucsi96]

I can confirm the previous comment. Setting network-concurrency doesn't help. Switching to Phone hotspot fixed the issue for me. My environment: Windows 10 (Linux subsystem - Ubuntu)

[sauntimo]

I'm on WSL and had been seeing this issue with the geo-tz package which has a (slightly odd) deeply nested folder structure. I tried some --network-timeout and --network-concurrency things but got nowhere. However, when I enabled long paths on Windows (see this SuperUser post) it's now working fine. Perhaps this might help some other people on WSL. Edit it looks like I spoke too soon. It was working, and linking dependencies is faster, but now I'm seeing the same error again.

[kunokdev]

Still breaking CI....

[jkonowitch]

Ditto - This is still an issue, even in 1.14

Arguments: 
  /home/jeff/n/bin/node /usr/share/yarn/bin/yarn.js install

PATH: 
  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/jkono/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/jkono/AppData/Local/hyper/app-2.1.2/resources/bin:/mnt/c/Users/jkono/AppData/Local/Programs/Microsoft VS Code/bin:/home/jeff/n/bin

Yarn version: 
  1.14.0

Node version: 
  10.15.1

Platform: 
  linux x64

Trace: 
  Error: ENOENT: no such file or directory, scandir '/mnt/c/Users/jkono/dev/PROJECT/node_modules/@storybook/addon-links/src'

Also:

➜  yarn cache dir
/mnt/c/Users/jkono/home/.cache/yarn/v4

[jwaldrip]

This is pretty annoying, we see this daily on local and ci machines.

[arichiardi]

Confirming this is happening all the time in CI for us as well

[entersb]

Hi, confirming that this issue happens on our CI. This is line with problem.

error https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/usr/local/share/.cache/yarn/v4/npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636/node_modules/core-js/es7/regexp.js'" info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[FezVrasta]

The same issue started happening today to one of our open source projects.

You can see a failing build here: https://travis-ci.com/quid/refraction/builds/103692106

And one that succeeds (with --network-concurrency 1) here: https://travis-ci.com/quid/refraction/builds/103693682

I hope it can help to diagnose the issue!

The source code of the repository is at: https://github.com/quid/refraction