Linking dependencies is taking a long time

[sfabriece]

Do you want to request a feature or report a bug? bug What is the current behavior? when installing a dependency, the third step: linking dependencies is taking a long time, even for a single package If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?

Please mention your node.js, yarn and operating system version. node: 6.7.0 OS: Windows 10

[macdja38]

I'm having linking dependancies take 200+ seconds with this https://github.com/macdja38/pvpsite/blob/master/package.json on Windows 10, off an SSD with a decent i7.

[macdja38]

It seems the performance issue may be caused by windows Defender, disabling it while possibly inadvisable reduced 200s to somewhere closer to 50.

[sfabriece]

I think that there should be a better solution than reducing security.

[macdja38]

Some other users have confirmed that disabling it just in the root directory of your project works, but I can't confirm as Windows Defender broke itself a bit when I tried to do that.

[wclr]

I've got the same with issue with git repo with about 30 dependencies. Windows 10 node v5.5.0 yarn 0.16.1

Disabling Windows Defender reduced linking time significantly

[wclr]

Probably should be "solved" by this PR?

[Daniel15]

Yeah there's not a lot we can do here unfortunately :( Virus scanners scan all files, and the npm ecosystem has a lot of small files. Small files generally have a bit more overhead on NTFS compared to other file systems such as EXT4 or ZFS, but it's exacerbated by virus scanners.

Having said that, Yarn should still be faster than npm, it just won't be as fast as on Linux or Mac.

[sharat]

I am on a Mac without having any antivirus scanners installed. But I still see the same problem, linking taking considerable amount of time even with a simple angular-js app.

[bjackson]

I have this issue too. It took 174s on Ubuntu.

[alanpog]

I started having this problem only after I upgraded from 0.17.8 to 0.17.19. Mac without antivirus.

[asolopovas]

Strange thing as well is that I need to go throw linking process each time when I delete a package as well. Npm does that faster. And linking really takes long time.

[hunterloftis]

This can be reproduced with this package.json (on Heroku):

{
  "name": "yarn-link-slowness",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "axios": "^0.15.3",
    "lodash": "^4.17.2",
    "react": "^15.4.1",
    "react-dom": "^15.4.1",
    "react-player": "^0.12.1",
    "react-redux": "^4.4.6",
    "react-router": "^3.0.0",
    "react-router-redux": "^4.0.7",
    "react-scripts": "^0.8.4",
    "redux": "^3.6.0",
    "redux-auth-wrapper": "^0.9.0",
    "redux-logger": "^2.7.4",
    "redux-promise-middleware": "^4.2.0",
    "redux-thunk": "^2.1.0"
  },
  "engines": {
    "node": "7.2.1",
    "yarn": "0.17.8"
  }
}

With yarn 0.17.8, the install takes 37s. With yarn 0.17.10, the install takes 97 seconds. No other changes (new Heroku apps each time).

[aj0strow]

✨ Done in 45.10s.

    "autoprefixer": "6.3.6",
    "babel-core": "6.7.6",
    "babel-jest": "13.0.0",
    "babel-loader": "6.2.4",
    "babel-plugin-transform-class-properties": "6.9.1",
    "babel-plugin-transform-object-rest-spread": "6.8.0",
    "babel-preset-es2015": "6.6.0",
    "babel-preset-react": "6.5.0",
    "bluebird": "3.3.5",
    "cardmask": "github:aj0strow/cardmask#v1.0.0",
    "chai": "3.5.0",
    "classnames": "2.2.5",
    "copy-webpack-plugin": "2.1.3",
    "core-js": "2.4.1",
    "css-loader": "0.23.1",
    "enzyme": "2.3.0",
    "file-loader": "0.8.5",
    "force-case-sensitivity-webpack-plugin": "0.1.1",
    "jest": "13.0.0",
    "jest-cli": "13.0.0",
    "json-loader": "0.5.4",
    "lodash": "4.11.1",
    "moment": "2.13.0",
    "ms": "0.7.1",
    "node-sass": "3.4.2",
    "postcss-loader": "0.9.1",
    "raw-loader": "0.5.1",
    "react": "15.2.0",
    "react-addons-css-transition-group": "15.2.0",
    "react-addons-test-utils": "15.2.0",
    "react-css-transition-replace": "2.0.1",
    "react-dom": "15.0.1",
    "react-redux": "4.4.5",
    "react-router": "2.3.0",
    "react-textarea-autosize": "4.0.3",
    "recompose": "0.20.2",
    "redux": "3.5.1",
    "redux-actions": "0.10.0",
    "redux-thunk": "2.0.1",
    "reselect": "2.5.3",
    "sass-loader": "3.2.0",
    "sinon": "1.17.4",
    "style-loader": "0.13.1",
    "webpack": "1.13.0",
    "webpack-dev-server": "1.14.1",
    "whatwg-fetch": "1.0.0",
    "zxcvbn": "4.3.0"

[Kreozot]

Please, can anyone explain what yarn does in "Linking dependencies" step exactly? Because max number on this step vary from ~1000 to ~65000 for same project on different machines. What does this number mean?

[mpseidel]

Having this issue as well. Adding a dependency with yarn add seems to trigger "Linking dependencies" and it takes forever. Had to switch back to npm for now.

node: 6.9.2 OS: Windows 10

[OlegLustenko]

node: 7.3.0 OS:Windows 10 64 Same for me

[RichiCoder1]

Same here. Links 23420...something, and takes about one and a half minutes on a good day.

[echobops]

Yarn 0.19.1 NodeJS 7.3.0 Windows 10

yarn add moment takes 105 seconds. It has no dependencies. :/

EDIT: Turning off Windows Defender does reduce time to ~30 to ~50 seconds. I tried excluding the directory I'm working in, but that didn't seem to help.

[JeffBaumgardt]

Just ran a fresh copy of create-react-app and it took 876.37s. I understand you don't have much/any control over how anti-virus works but my experience with NPM and CRA was much faster on Windows.

[wclr]

On windows 10 just use Ubuntu bash shell as a general advice.

[Daniel15]

On windows 10 just use Ubuntu bash shell as a general advice.

Disk I/O is extremely slow in Windows Subsystem for Linux, it's a known limitation at the moment

but my experience with NPM and CRA was much faster on Windows.

@JeffBaumgardt - Interesting... Yarn is slower on Windows, but it should still be faster than npm!

[SleeplessByte]

@Daniel15 it probably should be, but it's not. Node installs and de-installs were smaller for me. So I would do npm add <packages> --save-dev, delete yarn.lock and run yarn and that be fast than running yarn add <packages> -D a single time. Now hat everyone is on yarn, of course I don't want to delete the lock and force everyone to upgrade their bundle. Instead the following has been great:

cc @echobnet

For anyone on windows 10 + windows defender

1. First click settings

   

2. Scroll down to exclusions

   

3. Run yarn cache dir to get the location of your cache folder

   • Add this cache folder to the exclusion list
   • Add your project node_modules folder to the exclusion list

4. Speed-up for a react project x 3-10

[kybarg]

@SleeplessByte or you can simply add yarn and node to excluded processes.

[jribe]

Not just a problem on windows. I've been seeing horrendous link times on my Mac Pro.

OS: OS X 10.11.6 (El Capitan) Node: 7.6.0 Yarn: 0.20.3

[donaldpipowitch]

I can confirm that it is very slow on Mac 10.12.3, too. Not related to windows.

And it seems my setup is way slower than others in this thread. yarn sometimes tries to link around 600.000 files in small projects. This can take more than 30 minutes. I currently try it with a clean cache and nightly (v0.22.0-20170226.1626). I use the official registry as well as a custom private registry for certain scoped packages. However my colleagues don't suffer from this problem, so I don't think the custom private registry is the problem (and fetching packages is already finished anyway). We also have some relative files with path: protocol in our package.json.

I have a lot of problems installing https://github.com/google/material-design-icons which has a lot of small files which seem to be troublesome for other people as well (https://github.com/google/material-design-icons/issues/518). I don't know if my hardware is broken handling a lot of small files or something like that or if this is related at all. Again my colleagues don't have as much problems installing https://github.com/google/material-design-icons as I do.

Update

I'm not sure... it looks like installing a package with file: puts a module in the cache containing node_modules/ and other stuff. This is really a problem if you have multiple examples all containing their own node_modules/. It seems .npmignore, etc. is ignored for file: installs. This probably boils down to https://github.com/yarnpkg/yarn/issues/2165, if the solution is to not cache locally resolved files at all. If I open my cache ($ yarn cache dir) and look for modules why where install with file: and they contain a node_modules directory or other big directories, I can speed up the linking phase by removing these directories manually. Now everything seems to install with good speed.

[kuncevic]

[3/4] Linking dependencies...
Done in 947.71s. 

Had to wait this amount of time adding any new package with yarn add ... Win7 /w Yarn v0.21.3 Got material-design-icons package in my app. I think this one is related https://github.com/yarnpkg/yarn/issues/990

[asolopovas]

@kuncevich everything works fine on my end, during runtime of you yarn add start task manager ctrl + alt + esc check which program uses the highest cpu, in my case it was antivirus so I had to exclude not only the %appdata% directories but also the the project directory and things became fast again

[vbfox]

@kuncevic you might be affected by the bug I found on windows : https://github.com/yarnpkg/yarn/pull/2958

Essentially yarn can always copy every file in node_modules for every operation.

[kuncevic]

@asolopovas in my case it is node.exe like 10-26 %:

My AV is not an issue even if I just turned it off completely I cant see any yarn speed improvements.

node -v 6.9.2

[asolopovas]

@kuncevic update your node to 7 and check if that makes things faster, otherwise @vbfox is pointing in right direction.

[danpalmer]

Essentially yarn can always copy every file in node_modules for every operation.

@vbfox could you elaborate as to why? I'm looking at the verbose output of yarn and finding that almost all of the time is spent creating directories and copying files, as it seems to do each one individually, rather than, say, creating a directory for each package and then copying the whole package, which should be a fair bit faster, or even just symlinking all the packages which might be faster again. Are these not safe to do?

[vbfox]

@danpalmer the linking phase works in essentially 3 big steps:

1. Find every file that need to be in node_modules
2. Check this list versus what is already there and find what need to be copied around from cache to node_modules
3. Do the copy

Due to a libuv/nodejs bug (utime is used by yarn and the bug is that it set the milliseconds to zero) files copied by yarn in a previous run are always found to be different (Version in cache have a normal modified time but all files in node_modules have a zero-for-milliseconds version) so phase 2 always report that everything changed.

As the bug is fixed in node 7.1 it's pretty easy to fix if you aren't limited to LTS (First yarn operation on a repo will be slow as the files were created with the buggy utime but all following will be a lot faster). My PR essentially fix this by ignoring the milliseconds part of file times on windows when comparing them.

Regarding copying whole package it's not an operation that exists on current filesystems AFAIK they all work at the file level. The best windows provide is a FileCopy API (I have a PR to use it in yarn: https://github.com/yarnpkg/yarn/pull/2960) but it's just a little bit faster than using the native nodejs stream API.

As for symlinking I don't know why it isn't done, i'm not knowledgeable enough on javascript package managers (A few modifications are done to the package files like removing test folders but symlinking individual files wouldn't make that different) but as it isn't the case on linux/macos either (where it's a lot more common than on windows) there might be a good reason.

[kuncevic]

My experiment with upgrading to Node 7.8.0: https://github.com/yarnpkg/yarn/issues/990#issuecomment-290288375

[gravitypersists]

1. Find every file that need to be in node_modules
2. Check this list versus what is already there and find what need to be copied around from cache to node_modules
3. Do the copy

Considering that most of the time people are relinking swaths of libraries when switching between branches - maybe there's a better way to go about this?

Could you create a unique ID for each build of node_modules and then sym link the entire directory from a cache? This way, switching branches back and forth is really just symlinking different node_modules

Granted, you'll be writing a lot to disk since now you're caching every version of node_modules you run up against, but could there be optimizations there to sym-link to directories so that you're really only storing a tree of sym-links?

Forgive me for being naive, I'm not the most educated when it comes to the unix and even-less-so Windows file systems, but I'd be happy to dig into this, as an educational exercise, and try and provide a proof of concept of this idea if it's not obviously flawed in any way.

[laggingreflex]

pnpm and ied employ some of the techniques you mention I think, not entirely sure, tried them some time ago but they either caused issues on Windows or were not fast as yarn.

[sferrka]

Me too just had very long time to create-react-app with yarn windows server 2012 node 7.9.0 yarn 0.22 Done in 554.08s.

But in other cases it is much faster if no React installations included

[sharat]

I don't observe long linking time recently. Running Yarn - v0.23.2 node - 6.10.2 or 7.9.10 (using nvm to switch)

Tried this on a mac and archlinux (Manjaro)

[Morphexe]

I can confirm that adding node and yarn to the Windows Defender exclusions reduced the Linking time in around 60% on my windows machine.

[MattMorgis]

+1

[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 180.22s.

[JasonGoemaat]

Switching to node 7.9.0 didn't speed it up for me. Adding 'yarn', 'node' and 'npm' to windows defender (with and without .exe extensions, not sure what is required) did speed it up 3x for me, but still 50% longer than npm install.

Also removing all protection from anything running in node or any packages being installed doesn't seem like a good idea to me...

[dsschnau]

Adding my experience - adding node.exe/yarn.exe to windows defender's exception list cut our yarn install time in half (from 60s to 30s).

[joelsand]

I'm seeing this as well, makes it frustrating to iterate quickly while developing a package because it takes so long to update a single package.

[FrankFang]

yarn install v0.24.5
[1/4] Resolving packages...
[2/4] Fetching packages...
warning fsevents@1.1.1: The platform "linux" is incompatible with this module.
info "fsevents@1.1.1" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 338.20s.

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:    14.04
Codename:   trusty

  "dependencies": {
    "autoprefixer": "^6.7.7",
    "axios": "^0.16.1",
    "babel-core": "^6.24.1",
    "babel-loader": "7.x",
    "babel-preset-env": "^1.4.0",
    "coffee-loader": "^0.7.3",
    "coffee-script": "^1.12.5",
    "compression-webpack-plugin": "^0.4.0",
    "css-loader": "^0.28.0",
    "element-ui": "^1.3.3",
    "extract-text-webpack-plugin": "^2.1.0",
    "file-loader": "^0.11.1",
    "glob": "^7.1.1",
    "js-yaml": "^3.8.3",
    "node-sass": "^4.5.2",
    "path-complete-extname": "^0.1.0",
    "postcss-loader": "^1.3.3",
    "postcss-smart-import": "^0.6.12",
    "precss": "^1.4.0",
    "rails-erb-loader": "^5.0.0",
    "rails-ujs": "^5.1.0",
    "sass-loader": "^6.0.3",
    "style-loader": "^0.16.1",
    "turbolinks": "^5.0.3",
    "vue": "^2.3.0",
    "vue-loader": "^12.0.2",
    "vue-router": "^2.5.3",
    "vue-template-compiler": "^2.3.0",
    "webpack": "^2.4.1",
    "webpack-manifest-plugin": "^1.1.0",
    "webpack-merge": "^4.1.0"
  },
  "devDependencies": {
    "element-theme": "*",
    "element-theme-default": "^1.3.3",
    "eslint": "^3.19.0",
    "eslint-config-airbnb": "^14.1.0",
    "eslint-plugin-import": "^2.2.0",
    "eslint-plugin-jsx-a11y": "^4.0.0",
    "eslint-plugin-react": "^6.9.0",
    "nodemon": "^1.11.0",
    "webpack-dev-server": "^2.4.5"
  }

:(

[d42ohpaz]

Adding in my +1 on Mid-Summer 2010 MacBook Pro (Sierra 10.12.4) using yarn 0.24.5, node 7.10.0, and npm 4.2.0:

λ yarn add bootstrap-sass yarn add v0.24.5 [1/4] 🔍 Resolving packages... [2/4] 🚚 Fetching packages... [3/4] 🔗 Linking dependencies... [4/4] 📃 Building fresh packages... success Saved lockfile. success Saved 1 new dependency. └─ bootstrap-sass@3.3.7 ✨ Done in 123.52s.

"dependencies": {
    "@angular/animations": "^4.1.3",
    "@angular/common": "^4.0.0",
    "@angular/compiler": "^4.0.0",
    "@angular/core": "^4.0.0",
    "@angular/forms": "^4.0.0",
    "@angular/http": "^4.0.0",
    "@angular/material": "^2.0.0-beta.5",
    "@angular/platform-browser": "^4.0.0",
    "@angular/platform-browser-dynamic": "^4.0.0",
    "@angular/router": "^4.0.0",
    "bootstrap-sass": "^3.3.7",
    "core-js": "^2.4.1",
    "font-awesome": "^4.7.0",
    "material-design-icons": "^3.0.1",
    "materialize-css": "^0.98.2",
    "rxjs": "^5.1.0",
    "zone.js": "^0.8.4"
},
"devDependencies": {
    "@angular/cli": "1.0.1",
    "@angular/compiler-cli": "^4.0.0",
    "@types/jasmine": "2.5.38",
    "@types/node": "~6.0.60",
    "codelyzer": "~2.0.0",
    "jasmine-core": "~2.5.2",
    "jasmine-spec-reporter": "~3.2.0",
    "karma": "~1.4.1",
    "karma-chrome-launcher": "~2.0.0",
    "karma-cli": "~1.0.1",
    "karma-coverage-istanbul-reporter": "^0.2.0",
    "karma-jasmine": "~1.1.0",
    "karma-jasmine-html-reporter": "^0.2.2",
    "protractor": "~5.1.0",
    "ts-node": "~2.0.0",
    "tslint": "~4.5.0",
    "typescript": "~2.2.0"
}

[iwarner]

Switching back to npm install fixed it for me.. I was getting - u'stream': u'[3/4] Linking dependencies in yarn and no error in NPM..

Something has gone wrong with the latest build perhaps

Running this through Docker.

[FrankFang]

@iwarner npm 5.0 is a good choice.

[pharazon]

I run yarn in Vagrant (Ubuntu Xenial) and by Jenkins. There are two sub-projects with package.json. npm -v 3.10.10 node -v 6.10.1 yarn install v0.21.3

We switched from npm to yarn some time ago, because we had a timeout problem (4hours was not enough) for npm install.

Now yarn works about 30% of the time, but for 70% of the time we get 4h timeout at some point. It might the first yarn install, or the second, or we might timeout while running unit tests (jest).

[bestander]

This is a duplicate of https://github.com/yarnpkg/yarn/issues/990, there is a comparison chart and seems like latest versions of Yarn made some good progress there. If it is still a problem please file a new issue with repro steps and a comparison with latest npm

[rajat1saxena]

success Saved lockfile.
Done in 1737.79s.

Ubuntu 16.04 i5, 8 GB RAM

:(

[Benno007]

Windows 10 v 1709 + SSD + PowerShell + Node 6.12.2: Yarn install was fast until the last package, seemed to get stuck on a preinstall command. Followed the instructions here to add exclusions for windows defender, but also I had my source checked out to %USERPROFILE%\source which slowed it down drastically. Checked out in c:\ it was heaps faster.