search

yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry
https://classic.yarnpkg.com
Other
41.44k stars 2.72k forks source link

How to upgrade indirect dependencies? #4986

Open chinesedfan opened 6 years ago

chinesedfan commented 6 years ago

Do you want to request a feature or report a bug?

Feature.

What is the current behavior? yarn upgrade ignores indirect dependencies, so users can't upgrade them in yarn.lock. If I missed something, please tell me.

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior? yarn upgrade also supports indirect dependencies.

Please mention your node.js, yarn and operating system version. Node 8.9.0 yarn 1.3.2 OSX 10.12.6

milesj commented 6 years ago

@chinesedfan Have you tried yarn upgrade-interactive?

chinesedfan commented 6 years ago

@milesj Yes, it results in the same result and I also updated reproduce steps in issue description.

rally25rs commented 6 years ago

That is because yarn add is-alphanumerical@1.0.0 sets your package.json to exactly version 1.0.0 as you requested.

yarn upgrade respects your package.json semver range, and since you specified exactly version 1.0.0, it won't offer to upgrade to other versions.

You could resolve this a couple ways:

Edit:

Sorry, I just noticed there are different package names. alphanumerical and alphabetical look the same at a glance :)

Right, since there is no upgrade for is-alphanumerical, the dependency tree isn't traversed any deeper to handle its transitive dependencies.

You could try adding the --force flag and see if that makes it the subdependencies. Otherwise I think you are right, there isn't an easy way to do that other than yarn remove is-alphanumerical and yarn add is-alphanumerical

chinesedfan commented 6 years ago

@rally25rs Thanks for your reply! I tested 2 more cases.

OneCyrus commented 6 years ago

yes, this is a major problem with yarn at the moment. and it's already in discussion at #2394

rally25rs commented 6 years ago

duplicate #2394

AlexWayfer commented 6 years ago

Please, re-open: it's not duplicate.

2394 describes duplicating of meck-test-bb package (indirect dependency):

I got two copies of meck-test-bb

This issue describes just ability to upgrade indirect dependency (somehow).

chinesedfan commented 6 years ago

@rally25rs Forgive me to ping.

AlexWayfer commented 6 years ago

@rally25rs, please, you've closed both non-duplicating issues, it's wrong. Give us ability to upgrade indirect dependencies, please!

rally25rs commented 6 years ago

Sorry, there was some confusion over on the other issue. I originally thought 2394 was asking for a way to upgrade a transitive dep using a --deep flag, or something like that, so I had marked this issue as a duplicate of that. Later on after re-reading 2394 I think it was about something different, which is not a duplicate of this like I originally thought.

alex-thewsey-ibm commented 6 years ago

+1 for this feature request. Also an example for anybody dumb like me who needs to upgrade a specific indirect dependency manually in the interim:

Given explicit dependency jsonwebtoken has resolved implicit dependency jws^3.0.0 to vulnerablejws=3.1.4: and you need it to instead resolve to patched 3.1.5:

Delete the jws entry e.g. below from yarn.lock, and re-run yarn. The indirect dependency and any affected packages will be updated, without touching other things (on yarn v1.3 at least)

jws@^3.0.0, jws@^3.1.4:
  version "3.1.4"
  resolved "https://registry.npmjs.org/jws/-/jws-3.1.4.tgz#f9e8b9338e8a847277d6444b1464f61880e050a2"
  dependencies:
    base64url "^2.0.0"
    jwa "^1.1.4"
    safe-buffer "^5.0.1"

Edit: Punctuation

mkutny commented 6 years ago

@alex-thewsey-ibm, thanks for the workaround!

Worked on yarn v1.7.

Subtletree commented 6 years ago

ty, worked Yarn 1.9.2

joelpurra commented 6 years ago

Might help to nudge yarn with selective dependency resolutions, even if it's for a single dependency. Thanks to @remolueoend for the hint! https://yarnpkg.com/lang/en/docs/selective-version-resolutions/

From the docs:

{
  "name": "project",
  "version": "1.0.0",
  "dependencies": {
    "left-pad": "1.0.0",
    "c": "file:../c-1",
    "d2": "file:../d2-1"
  },
  "resolutions": {
    "d2/left-pad": "1.1.1",
    "c/**/left-pad": "1.1.2"
  }
}
ai commented 5 years ago

We need this feature in Autoprefixer to suggest users how to update caniuse-lite in their yarn.lock https://github.com/postcss/autoprefixer/issues/1184

richseviora commented 5 years ago

Same problem here. I expected yarn upgrade caniuse-lite browserslist to upgrade the sub-dependency. It didn't do that, nor did it give me an error message saying that it can't upgrade it b/c it's not a dependency.

Deleting the relevant lockfile entries and then re-running yarn as @alex-thewsey-ibm suggested fixed the immediate issue for me.

jrochkind commented 5 years ago

It is odd to me that yarn is missing tnis feature. I am new to yarn (and npm), so assumed there must be a way to do this. I'm still not completely sure if there is a non-obvious way to do this that none in this thread know, or if there's really no way to do this.

If there's really no way to upgrade a transitive/indirect dependency in the lockfile without adding it to package.json... I don't understand how yarn users do without it.

damaon commented 5 years ago

This is unexpected behaviour IMO - I tried to update lodash recently with yarn upgrade lodash@4.17.15 and it still haven't updated for any transient dependency.

I was still left with all major(^4.X.X) and patch(~4.17.X) versions pointing to old version.

Only way to fix it is via manual editting of yarn.lock and then maybe running yarn upgrade to consolidate changes. I would expect a little better from such widely used tool.

Is this acknowledged bug or yarn is conservative by default and I'm expected to manually edit yarn.lock or use some flag?

cheshrkat commented 5 years ago

I suspect I have the same security alert to resolve as @Machiaweliczny ;) It would be really useful to override the indirect dependency while we wait on projects to fix their own. Even with highly responsive projects there's a delay waiting for fixes and releases.

djmitche commented 5 years ago

Indeed, this is problematic for security issues in indirect dependencies, and the workaround of editing yarn.lock, while effective, is disappointing and difficult to automate. If this is not the default behavior for some reason, it'd be great to add an option like --include-indirect that will force Yarn to follow the indirect dependencies.

jrochkind commented 5 years ago

I don't think it should need an option, I don't see why yarn upgrade [an indirect dependency] doesn't just update the yarn.lock to latest version of the indirect dependency that is allowed by dependency tree, without any need for additional options. I think right now it's just a no-op?

However, another workaround I found I am happy with is adding resolutions to my package.json. If lodash is an indirect dependency, and I know that I need it to be >= 4.7.13 to avoid a security vulnerability, I can add to my package.json:

  "resolutions": {
    "lodash": ">= 4.17.13"
  }

Then just run yarn install, it will update the yarn.lock to meet that requirement, or complain if it can't because it conflicts with an indirect dependency.

This actually seems to have worked pretty well in my case; I wonder if it's not a "workaround" but the intended solution? It took me a while to discover though. And I don't understand things well enough to be sure this is a universal/correct solution or if there might be any problems in some cases with it. If it is the 'right' solution for upgrading indirect dependencies, it was somewhat hard to find.

milesj commented 5 years ago

Why not yarn install the transitive dep you want to update but don't commit the package.json changes, just the yarn.lock?

djmitche commented 5 years ago

Why not yarn install the transitive dep you want to update but don't commit the package.json changes, just the yarn.lock?

I don't think that will (always?) work, because yarn will happily install multiple versions of the same package, even if a single version would satisfy all of the relevant semver ranges. So this would install the version you want, but not remove the version you don't want.

milesj commented 5 years ago

@djmitche Right. You would need to install the version within the range expected. Not ideal and a bit tedious, but an available stopgap for now.

kumar303 commented 5 years ago

@djmitche Indeed, this is problematic for security issues in indirect dependencies, and the workaround of editing yarn.lock, while effective, is disappointing and difficult to automate.

This is another workaround which is slightly more automatable:

yarn remove is-alphanumerical
yarn add is-alphanumerical

Using the example in the PR description, this would remove the top level dep then re-add it which will get all of its latest sub-deps, according to the ranges specified by is-alphanumerical (caret ranges, for example). It will then produce a new lock file.

The side effect is that it will update all sub-deps which is not ideal. For a security issue in sub-dep A I'd want to only update sub-dep A.

The workaround of adding sub-dep A as a direct dep just to fix a security issue is worse because it creates confusion (the package is not used directly) as well as a maintenance burden.

Cellule commented 4 years ago

yarn remove is-alphanumerical yarn add is-alphanumerical

This seems to be the only reliable way to actually update a dependency. I realized today that we were stuck on a 1.0.0 version of a package that had updated to 1.1.0 a year ago. The package we were using did use ^1.0.0 and all the times we "upgraded" that package it never picked up the new 1.1.0 version of its dependency. Turns out there was a pretty bad bug that were silently failing in our product that should have been fixed a year ago without me wasting a day investigating why we were having this issue.

I can't believe editing yarn.lock manually, removing then re-adding the package or using selective resolution are the "ways" to update an indirect dependency.

IMO, yarn upgrade should update the indirect dependencies to the latest accepted semver version, that's why we upgrade a package. I mean, if there's was a break in the semver range it would update the indirect dependencies, so it should do it always.

djmitche commented 4 years ago

Would it help to write some kind of external script to do this? I suppose it would just remove all yarn.lock entries for the given package, and then re-run yarn?

pftg commented 4 years ago

There is a simple script to do this: https://gist.github.com/pftg/fa8fe4ca2bb4638fbd19324376487f42

nnmrts commented 4 years ago

Can one of the maintainers please change the label from cat-feature to cat-bug?

rally25rs commented 4 years ago

Can one of the maintainers please change the label from cat-feature to cat-bug?

why? this isn't a bug. It is as designed. yarn upgrade was never intended to be used to upgrade a transitive dependency. The originally opened "issue" is even labeled as a feature request.

Internally upgrade uses yarn outdated to determine what is out of date and what versions to upgrade to. outdated only checks direct dependencies.

I could be wrong, or perhaps it has changed, but I'm pretty sure that npm upgrade at least as of 3 years ago when yarn upgrade was last reworked, also doesn't provide a way to upgrade a transitive dep. (again, that might have changed since over the years, I'm not too up-to-date on npm's changes).

Anyone is free to submit a PR to add this functionality. This is an open source project and it is up to the community at large to contribute. This feature request has been open for years but no one has stepped up to implement it and that is why it hasn't been "fixed".

pftg commented 4 years ago

@nnmrts could you check my script https://github.com/yarnpkg/yarn/issues/4986#issuecomment-562719589 will it help you for now?

nnmrts commented 4 years ago

@rally25rs Sorry, I was tired and grumpy, consider my comment as resolved. 😬

nnmrts commented 4 years ago

@nnmrts could you check my script #4986 (comment) will it help you for now?

That script unfortunately didn't work for me, tried it out yesterday. Maybe I did something wrong, but my whole yarn.lock file got "emptied" by the script.

dacioromero commented 4 years ago

I'm not sure how good of a workaround this is, but I ran this script that I wrote:

const fs = require('fs')
const lockfile = require('@yarnpkg/lockfile')
const package = require('./package.json')

const lock = lockfile.parse(fs.readFileSync('yarn.lock', 'utf-8')).object

const allDeps = new Set()

const parseDep = ([name, version]) => {
  allDeps.add(`${name}@${version}`)
}

Object.entries(package.dependencies).forEach(parseDep)
Object.entries(package.devDependencies).forEach(parseDep)

const newLock = Object.fromEntries(Object.entries(lock).filter(([dep]) => allDeps.has(dep)))
const newLockString = lockfile.stringify(newLock)

fs.writeFileSync('yarn.lock', newLockString)

Then ran yarn install and it seems to install the latest verisons of indirect dependencies.

ayushya commented 4 years ago

I was able to resolve deep/indirect dependencies successfully. I wonder when we will get an official support.

https://medium.com/@ayushya/upgrading-javascript-packages-deep-dependencies-using-yarn-8b5983d5fb6b

I have tried to resolve and explained the risks of re-generating the yarn.lock and have suggested what should be done.

Let me know if this works for you guys as well. Or any suggestions to improve the process of upgrading.

jrochkind commented 4 years ago

@ayushya Hm, that does seem to work, genius.

I wonder if yarn would accept a patch in which yarn upgrade to an indirect dependency (or some other command?) just... did that?

natematykiewicz commented 4 years ago

@jrochkind I would have expected a yarn upgrade of an indirect dependency to upgrade it even if it's not my direct dependency. Without that feature, you can be years behind on updates of indirect dependencies.

In my case, I was trying to upgrade fsevents, so that it didn't spew errors when I did a yarn install (https://github.com/fsevents/fsevents/issues/278). fsevents isn't a package I directly use -- it's something that webpack-dev-server uses. But shockingly, I was locked to whatever version existed when webpack-dev-server was first installed in this app.

I had to use this trick to upgrade it, which seemed like a total hack. https://github.com/yarnpkg/yarn/issues/4986#issuecomment-395036563

FelipeLujan commented 4 years ago

the workaround is not working for me, unfortunately, the old deep-dependencies were added again to the yarn.lock file after deleting them. I can see them back in the node_modules folder and yarn.lock file after running yarn install.

maybe the workaround is not compatible with yarn workspaces?

jrochkind commented 4 years ago

@FelipeLujan when the workaround works, the deep-dependencies are still added again to the yarn.lock file -- that is expected -- but with new later versions. But only with new versions if there are new versions released, and if they are allowed by the dependency tree. If some intermediary dependency expresses a restriction which doesn't allow an upgrade, they still can't be upgraded. They are just upgraded to the latest allowed by the restrictions in the tree.

i don't use yarn workspaces though, so can't say if that's relevant.

ayushya commented 4 years ago

@FelipeLujan AFAIK yarn workspaces work in a similar way.

The workaround to delete the package section will only upgrade the package to the latest MINOR/PATCH version. If you want to upgrade the Package to a newer MAJOR version you have to find the package dependency chain running yarn why package-name-here and upgrade the package at the top of its chain.

CAUTION: Do test your code as upgrading the packages to a newer MAJOR version might bring some breaking changes.

djmitche commented 4 years ago

https://github.com/djmitche/yarn-minify might help with this..

jrochkind commented 4 years ago

@ayushya's workaround is working great for me, manually editing the yarn.lock to remove an indirect dependency, then running yarn install to re-add it at a later version.

To me, this seems like a feature that should be built into yarn though, rather than requiring you to manually edit yarn.lock. It seems like it should be fairly striaghtforward to make a feature that is as if you had manually edited to delete the dependency and run yarn install. I feel this feature really oughta be built into yarn, and am kinda confused as to why it isn't.

ghost commented 4 years ago

I was able to resolve deep/indirect dependencies successfully. I wonder when we will get an official support.

https://medium.com/@ayushya/upgrading-javascript-packages-deep-dependencies-using-yarn-8b5983d5fb6b

I have tried to resolve and explained the risks of re-generating the yarn.lock and have suggested what should be done.

Let me know if this works for you guys as well. Or any suggestions to improve the process of upgrading.

I think that's the same resolution given by @alex-thewsey-ibm. Deleting that particular dependency from yarn.lock helped me. Anyways, Thanks for this hack☺️

felipecsl commented 4 years ago

Using resolutions in package.json worked for me https://github.com/webpack/webpack-dev-server/issues/2739#issuecomment-695164486

vfonic commented 4 years ago

This should at least return some warning:

yarn add is-alphanumerical@1.0.0
yarn upgrade is-alphabetical

This is what I get instead:

success Saved lockfile.
success Saved 0 new dependencies.

There are zero changes in package.json and yarn.lock even though there's a new is-alphabetical package version available.

mpdude commented 3 years ago

For the record, GitHub's Dependabot will propose updates also for transitive (indirect) dependencies if those contain known security issues.

https://dependabot.com/blog/securing-javascript-transitive-dependencies/

olivierlacan commented 3 years ago

@mpdude This is precisely why selective transitive dependency upgrading not being possible in Yarn (or NPM) is so problematic. We now have the largest code hosting service in the world (GitHub) recommending transitive dependency security updates that can't be achieved without complicated and error-prone manual manipulations of the lockfile or git history trickery.

This problem is only going to grow exponentially as more people get alerts from their repos and security scanning software (within companies) and business goals start prioritizing staying on top of attack vectors more and more.

There is a pattern for how to do this successfully: bundle update <dependency> will try to resolve a new dependency graph regardless of whether <dependency> is a direct or indirect dependency and regardless of the version resolved for it in the lockfile. Bundler even offers a --conservative flag to avoid indirect dependency updates.

I'm curious why yarn can't be made to follow the same pattern. I'm not trying to diminish the work involved, I'm sure it's far beyond my understanding since I'm about as inexperienced in graph traversal as anyone can be, but it's concerning to not see a strong focus on hashing out a plan to get this feature tackled and eventually added. Nearly half

shamilovtim commented 2 years ago

Is pinning all direct dependencies to their exact versions, doing yarn install and then deleting the lock file immediately afterwards, following by another yarn install not a good solution?

moroine commented 2 years ago

I don't know why https://github.com/djmitche/yarn-minify feature is not built-in yarn update command 🤔

djmitche commented 2 years ago

..because it's a hack :)

Implementing that within yarn itself, in an appropriate fashion, is much harder!