search

yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry
https://classic.yarnpkg.com
Other
41.44k stars 2.73k forks source link

yarn install fails with `ENOENT: no such file or directory` occasionally #2629

Closed NiGhTTraX closed 3 years ago

NiGhTTraX commented 7 years ago

Running yarn install as part of a build step for a Docker image based on node:7 fails on Travis CI with ENOTEMPTY, EEXISTS errors. It always seems to error on the webdriverio package.

yarn install v0.19.1
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/webdriverio/-/webdriverio-4.6.2.tgz: ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol/timeouts.js'".

When Travis runs yarn install as part of the install phase it works just fine. The error only happens when building a Docker image.

Repo which reproduces this issue.

node:7 OS: Docker + Travis CI yarn: 0.19.1 package.json yarn.lock

I've tried installing yarn both with npm install -g and with apt and both methods cause the failure on Travis.

Weirdly enough, the image builds successfully on my local machine which runs Ubuntu 16.04.1 LTS with Docker version 1.13.0, build 49bf474.

Daniel15 commented 7 years ago

Interesting, so it only fails on Travis, but works when testing locally? That's very weird given Docker is supposed to ensure the environment is consistent.

NiGhTTraX commented 7 years ago

@Daniel15 I know right...

I've downgraded node to version 6 and it still fails on Travis. I've added the --verbose flag to yarn install and all I got was

verbose Performing "GET" request to "https://registry.yarnpkg.com/spawn-wrap/-/spawn-wrap-1.3.4.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs/-/yargs-6.6.0.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-4.2.1.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/fibers/-/fibers-1.0.15.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/selenium-standalone/-/selenium-standalone-5.11.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/tcp-port-used/-/tcp-port-used-0.1.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/babel-runtime/-/babel-runtime-5.8.38.tgz".
verbose Error: ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'
    at Error (native)
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'".

I am open to ideas on how to debug this.

twooster commented 7 years ago

Downgrading to yarn 0.18.1 seemed to fix this for me. Seems like 0.19 might have a regression; see #1834

victornoel commented 7 years ago

I also have this problem with yarn 0.23.3, it's happening not when building an image but simply when running some CI. The error is the following:

$ time yarn --frozen-lockfile
yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/builds/linagora/petals-cockpit/yarncache/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src'".
info If you think this is a bug, please open a bug report with the information provided in "/builds/linagora/petals-cockpit/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

real    0m9.812s
user    0m7.596s
sys 0m0.932s

I think there may be some strange way of doing remove of files…

Important point: the cache was empty!

victornoel commented 7 years ago

And on my machine, if I try to repro, I get this:

yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: EEXIST: file already exists, mkdir '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src/metadata'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
victornoel commented 7 years ago

And with yarn 0.21.2:

yarn install v0.21.2
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/bundles/core.umd.js'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

That's terrible!

victornoel commented 7 years ago

And I concur with @twooster about 0.18.1 working ok!

victornoel commented 7 years ago

@Daniel15 it doesn't work locally either. Actually, it NEVER work when the cache is empty for me!

bestander commented 7 years ago

@victornoel the recent error could be https://github.com/yarnpkg/yarn/issues/2714

victornoel commented 7 years ago

@bestander I did try 0.19.1 at the time and it didn't work…

I retried, and now the bug:

So I don't think it is the same, let's hope you can reproduce it at least…

victornoel commented 7 years ago

(actually, I just tried again and reproduced the bug with an empty cache and yarn 0.21.2 while it wasn't the case before, maybe there is another file somewhere else that is the source of this problem, and that is not in the cache?)

victornoel commented 7 years ago

@bestander I will still test yarn as soon as #2744 is available as a nightly build :)

bestander commented 7 years ago

Ping me if I can help. The best action is to send a PR with a broken (and skipped) e2e test.

victornoel commented 7 years ago

@bestander well, nope, I still get errors such as:


➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/facade/lang.d.ts'".

or:

➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

I will see if I can make an e2e test.

victornoel commented 7 years ago

@bestander anyway I can get a complete stacktrace of the error?

I only see this in the yarn-error.log:

Trace: 
  Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.es5.js'
      at Error (native)

That's a bit useless :)

victornoel commented 7 years ago

the detailled error is:

{ Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js'
    at Error (native)
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_type: 'File',
  fstream_path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_class: 'FileWriter',
  fstream_stack: 
   [ '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/fstream/lib/writer.js:285:28',
     '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/graceful-fs/polyfills.js:284:29',
     'FSReqWrap.oncomplete (fs.js:123:15)' ] }

Not sure exactly what to do with this… it's happening at package-fetcher.js, line 56 exactly,  I have trouble finding the source though…

victornoel commented 7 years ago

It seems stupid, but I feel like it fails only when my networked npm mirror (a sonatype nexus in my company) has mirrored the @angular/core artefact. If it hasn't, things go well and then it fails on another artefact that is already mirrored (typescript in this case).

If I remove the artefact from the nexus mirror by hand, it works!

So… it is a bit like yarn can't keep up if things arrive too fast ^^ because when I use the normal npm registry without using my mirror, things usually goes well (we have a slow internet connection). And it would explain why it often fails on CI systems because they usually have very fast internet connections…

That's a bit of stretch to conclude that, but it could help find the origin of the problem. WDYT @bestander?

For the record, I think the error is emanating from tar.Extract step in the fetching pipeline, but I'm not totally sure ^^

bestander commented 7 years ago

Thanks for researching more, @victornoel, you might be on something here.

bestander commented 7 years ago

I can repro scenario from https://github.com/yarnpkg/yarn/issues/2629#issuecomment-282745896. Looking into it.

I get 

error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/Users/bestander/Library/Caches/Yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

However if I just try yarn install again and again it will eventually finish the installation. Looks like exploding the .tgz file ends with error.

Update:

bestander commented 7 years ago

Some help investigating why untaring those few dependencies (in my case typescript and angular-core) causes error is welcome. Concurrency? Bug in https://github.com/npm/node-tar?

bestander commented 7 years ago

@victornoel, can you reproduce the bug with yarn install --network-concurrency 1?

victornoel commented 7 years ago

@bestander with --network-concurrency 1 the bug does not appear (while without it, it appears every time). But what is the default value of this parameter? Whichever value I choose for it (1, 2, 4, 8), it works, while if I don't put it at all, it fails…

bestander commented 7 years ago

The default is 15, I can repro the issue with concurrency 15 with a clean checkout of https://gitlab.com/linagora/petals-cockpit.git#075bac4c54fee466568c000c7ffe8025f593e212.

victornoel commented 7 years ago

Excellent news! One step further toward a solution AND a workaround :)

bestander commented 7 years ago

Some results.

TL;DR I am out of ideas how to properly fix it for good, this needs some deeper Node.js knowledge.

  1. Network can be eliminated from possible issues. I've setup offline mirror for the .tgz files in yarn.lock and can reproduce the issue with packages installed from disk.

The issue is in unzip/untar stream in tarball-fetcher code.

  1. I tried a different library that extracts tar - https://github.com/mafintosh/tar-fs vs current https://github.com/npm/node-tar/. They both fail the same way. Going a bit deeper - exceptions are happening in node when doing multiple mkdirp operations 
    Error: ENOENT: no such file or directory, chmod '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts'
errno: -2,
code: 'ENOENT',
syscall: 'chmod',
path: '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts' }

I think core-4.0.0 and typescript-2.2.1 fail because they have quite a few files and deep folder structures, and they fail to install while making many concurrent mkdir/copy operations.

Every time there is a different syscall that fails: chmod, rmdir, mkdir, lstat, utime.

And it is not something obvious in the libraries' code.

  1. Fails the same on Node 4, 6 and 7.

  2. I could not reproduce the error with concurrency set to 8, so I'll send a PR to reduce default network concurrency.

  1. I was wondering how concurrency affects installation speed.

5.1. Using offline-mirror (no download), on my MBPro 13", clean cache and using node-tar to unzip files. Concurrency 12 - fails Concurrency 8 - 18 seconds Concurrency 4 - 18 seconds Concurrency 2 - 21 seconds

5.2. Using offline-mirror (no download), on my MBPro 13", clean cache and using tar-fs to unzip files. Concurrency 12 - 15 seconds Concurrency 8 - 15 seconds Concurrency 4 - 17 seconds Concurrency 2 - 18 seconds

5.3. Downloading packages from internet, on my MBPro 13", clean cache and using tar-fs to unzip files. Concurrency 12 - failed once Concurrency 8 - 21 seconds Concurrency 4 - 23 seconds Concurrency 2 - 34 seconds

Looks like setting concurrency to 8 is safe enough, also it makes sense to switch the tar library. I'll follow up with a PR.

bestander commented 7 years ago

A proper way to fix this would be forking https://github.com/mafintosh/tar-fs and doing smarter fs operations, e.g. using mkdir for every folder only once

victornoel commented 7 years ago

tar-fs maintainer seems to be active, maybe we could open an issue there and see what they know/propose about this?

bestander commented 7 years ago

@victornoel, would you do that please?

victornoel commented 7 years ago

@bestander done! mafintosh/tar-fs#61 :)

ProdigySim commented 7 years ago

I ran into this error message in a somewhat similar scenario when testing out yarn on my jenkins' build agents.

Do you know what the conditions required to trigger this bug are? I'd like to replace my build system's npm calls with yarn for speed, but if I have to disable concurrency I'm worried it might negate any bonus there.

victornoel commented 7 years ago

@ProdigySim, as explained in #2829 (which was merged in yarn master), reducing the network concurrency has no big effect on the performance of yarn. You can simply set it to 8 and it should be alright. Anyway, even when downloading 8 dependencies at a time, I'm not sure most drive is going to follow in throughput so you won't loose much for sure :)

ProdigySim commented 7 years ago

@victornoel Thanks for the information. I'm not sure just reducing --network-concurrency will be enough in my case, since we would also run multiple instances of yarn in parallel.

I can repro this issue even with --network-concurrency 1, but maybe I should open a separate issue for that?

Using the same test repo you gave above:

#!/bin/bash
set -x # echo commands

# Clear yarn cache
rm -rf $(yarn cache dir)

# Clone the repo into two separate spots
git clone https://gitlab.com/linagora/petals-cockpit.git repo1
git clone https://gitlab.com/linagora/petals-cockpit.git repo2

# Run yarn on both in parallel
cd repo1/frontend && yarn --network-concurrency 1 &
cd repo2/frontend && yarn --network-concurrency 1 &

This nets me the error every time (4 for 4 so far)

error An unexpected error occurred: "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.2.tgz: 
ENOENT: no such file or directory, lstat '/Users/<snip>/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.2-59535050e5d0e6141417186eee571296f8e9c3d0/@angular/core.es5.js'".

On yarn 0.21.3, node v4.5.0, OSX 10.11.6

So far I've been able to work around this issue by only including yarn on build jobs that will never run in parallel, or use completely different sets of packages, but even then I'm not sure it will be safe--hence asking about root conditions for this bug.

jklmli commented 7 years ago

@ProdigySim This is a separate, but related, issue caused by Yarn's global download cache. A workaround is to create a different cache for each directory.

You can still run with --network-concurrency 8. (I actually have no issues with unlimited network concurrency.)

More context here.

victornoel commented 7 years ago

@bestander surprisingly, today, the problem reappeared (triggered by a tar for a new version of angular ^^) even with the network concurrency at 8, but only on my CI… I moved it to 2 and it works (and I don't really care if it takes a few more seconds to download dependencies, so it's ok for now). We don't seem to get feedbacks from the tar-fs project… who else could we contact for help on that?

weedgrease commented 7 years ago

having this issue as well on my Travis builds for OS X. I've clearned the cache and set network concurrency but nothing has helped.

victornoel commented 7 years ago

@kevingelion to which value did you set the network concurrency? be drastic and set something like 2 to see if the problem is this one :)

weedgrease commented 7 years ago

@victornoel I set it to 1 and 2, and both options resulted in a fail. I used yarn --mutex network and no dice either.

victornoel commented 7 years ago

@bestander the following hack fixes (edit: NOT) the problem:

diff --git a/src/util/request-manager.js b/src/util/request-manager.js
index e0e134a2..995dac69 100644
--- a/src/util/request-manager.js
+++ b/src/util/request-manager.js
@@ -214,8 +214,7 @@ export default class RequestManager {
     }, params.headers);

     const promise = new Promise((resolve, reject) => {
-      this.queue.push({params, resolve, reject});
-      this.shiftQueue();
+      this.execute({params, resolve, reject});
     });

     // we can't cache a request with a processor

Obviously it is not a fix, it completely bypasses the request manager and its queuing system, but it shows that the problem is coming from this subsystem.

bestander commented 7 years ago

Thanks, Victor!

On 24 March 2017 at 18:07, Victor Noël notifications@github.com wrote:

@bestander https://github.com/bestander the following hack fixes the problem:

diff --git a/src/util/request-manager.js b/src/util/request-manager.js index e0e134a2..995dac69 100644 --- a/src/util/request-manager.js +++ b/src/util/request-manager.js @@ -214,8 +214,7 @@ export default class RequestManager { }, params.headers);

 const promise = new Promise((resolve, reject) => {
  • this.queue.push({params, resolve, reject});
  • this.shiftQueue();

  • this.execute({params, resolve, reject}); });

    // we can't cache a request with a processor

Obviously it is not a fix, it completely bypasses the request manager and its queuing system, but it shows that the problem is coming from this subsystem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yarnpkg/yarn/issues/2629#issuecomment-289102067, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBdWF66L-NzAInx7Bhs6V7s7LKahxxUks5rpAZ1gaJpZM4L3JbX .

victornoel commented 7 years ago

whoups not, it doesn't :D but it does improve things a bit

victornoel commented 7 years ago

Sorry for the false positive, I was too eager to report my findings :)

It changes things because before I could run yarn many times and it would never succeed downloading the angular-core dependency or the typescript (always these ones), but there it failed the first time, and succeeded the second time, and I forgot to remove the cache in between my tries so I thought it was working.

victornoel commented 7 years ago

Well it depends, now sometimes it works, sometimes it does not (with the hack, so it's not a total miss or my internet connection is too slow right now...)

kaicataldo commented 7 years ago

I'm running into this too in our CI builds. After a lot of testing, I've also finally been able to reproduce locally.

Again, it does sometimes work, but it fails often with either of the following errors (which makes it seem like there's some kind of race condition somewhere):

I've isolated it to one package that we install directly from a private repository on GitHub. Interestingly enough, the package being referenced in the error message is always a dependency of this package (and it is always another package we install directly from GitHub, though not a private repository). So it seems like one repro case is installing packages from private GitHub URLs who have subdependencies that are also installed from GitHub repositories (not necessarily private).

Not sure if this helps at all...I'm happy to try to help out in any way I can!

Edit: Not sure if this is helpful or not, but the top level package is listed in the format of "git+ssh://git@github.com/org/package.git#v1.0.0", and in the error, the subdependency being downloaded looks like it's being downloaded over https with a url of "https://codeload.github.com/org/package/tar.gz/ljasdf08i234098aifj".

bestander commented 7 years ago

I am investigating this a bit more. Trying to build a standalone script for concurrent tar-fs extracts but I tend to think that the problem is in tar file breaking during download.

bestander commented 7 years ago

Found it, Doh.

In the example from https://github.com/yarnpkg/yarn/issues/2629#issuecomment-282745896 Yarn has duplicate packages that are getting downloaded and extracted in parallel. The duplicated ones are @angular/core/-/core-4.0.0-rc.1 and typescript/-/typescript-2.2.1.tgz.

With high concurrency we just happen to do concurrent extraction into the same cache folder. I'll investigate why Yarn does not deduplicate those two pacakages and send a fix.

No magic on OS or tar extraction level.

victornoel commented 7 years ago

haha, good job @bestander, glad we finally found the problem!

kaicataldo commented 7 years ago

Awesome work @bestander :tada:! Running into both https://github.com/yarnpkg/yarn/pull/3090 and https://github.com/yarnpkg/yarn/pull/3106 was what blocked us from using Yarn.

victornoel commented 7 years ago

thanks!

skylize commented 7 years ago

I had this problem installing prop-types module. Each time I tried to install it would ENOENT on a different filename. For me the problem went away after installing npm 5.0.2

$ yarn add prop-types
yarn add v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/prop-types/-/prop-types-15.5.10.tgz: ENOENT: no such file or directory
....

$ npm -g install npm

# whoops, looks like npm installed itself to different location than apt-get did
$ npm -v 
3.5.2

# remove the cached link from shell so the right version can surface
$ hash -d npm
$ npm -v
5.0.2

$ yarn add prop-types
... properly installs prop-types as expected
Daniel15 commented 7 years ago

@skylize That's likely to be coincidental - Yarn doesn't use the npm client at all, so the version of npm does not affect execution of Yarn at all.