Updating only one locked dependency

[k4nar]

Sometimes I'm doing a PR and I want to update a specific dependency but I don't want to deal with updates of all my dependencies (aiohttp, flake8, etc…). If any breaking change was introduced in those dependencies, I want to deal with it in another PR.

As far as I know, the only way to do that would be to pin all the dependencies that I don't want to update in the Pipfile. But I find it to defeat the purpose of Pipenv in the first place :) .

So my feature request would be to be able to do something like:

$ pipenv lock --only my-awesome-dep

That would generate a Pipfile.lock with updates for only my-awesome-dep and its dependencies.

I can probably make a PR for that, but I would like to get some feedback first.

[k4nar]

That could also be useful for pipenv install, as sometimes I want to install a new dependency without updating others.

[vphilippon]

There's a little thing to take into account here: Changing a single dependency could change the overall set of requirements. Ex: Updating foo from 1.0 to 2.0 could require to update bar to >=2.0 (while it was <2.0 before), and so on.

I know that in the context of pip-tools itself (from which pipenv takes its dependency resolution algorithm), running the dependency resolution will only "update" the required packages when "re-locking" if there's an existing lock file. It does so by checking if the existing pins in the lockfile are valid candidate first when selecting candidate in the resolving. pipenv could probably do the same.

I think its a reasonable idea. Otherwise, if you want to update absolutely only one dependency, pipenv would have to have a mode to block if changing a dependency causes other changes, or else you would loose the guarantee of a valid environment.

I hope this helps!

[k4nar]

Indeed, that was what I meant by:

That would generate a Pipfile.lock with updates for only my-awesome-dep and its dependencies.

[brettdh]

Agree 100% - and I'll go a bit farther: this should be the default.

That is, pipenv install foo should never touch anything besides foo and its dependencies. And pipenv lock should certainly never upgrade anything - it should just lock what's already installed.

AFAICT, this is how npm, yarn, gem, etc. work; it makes no sense to have a lockfile that doesn't actually lock packages, but trusts package authors to not break things in patch releases, and therefore upgrades them without being asked. I can see the use of allowing upgrades, but that should be opt-in, since it's more surprising than not upgrading them.

I apologize if I'm hijacking this issue for something else, but since this is so closely related to an issue I was about to create, I thought I'd start the conversation here. Feel free to tell me I should make a new one.

[brettdh]

Just found this related issue as well: https://github.com/kennethreitz/pipenv/issues/418

Being able to specify pipenv install --upgrade-strategy=only-if-needed seems like what I'm looking for, though of course as I mentioned I think that should be the default, as it's becoming in pip 10. But being able to specify it semi-permanently via env var would be something, anyway.

I would be surprised if that change breaks anyone's workflow (famous last words), since it's more conservative than --upgrade-strategy=eager.

[brettdh]

Tried to work around this by setting export PIP_UPGRADE_STRATEGY=only-if-needed in my shell config. This doesn't work, and pipenv lock exhibits these surprising behaviors:

1. It "upgrades" packages that don't need to be upgraded (but...)
2. It actually doesn't upgrade the installed versions! i.e. pip freeze and Pipfile.lock show different versions!

Guessing pipenv is delegating to pip for the install, and pip respects its environment variable settings, but pipenv lock doesn't.

[techalchemy]

@k4nar What happens right now that you are finding undesirable? Because if you upgrade a dependency that has cascading requirements obviously it will have consequences for other dependencies. Are you suggesting some kind of resolver logic to determine the most current version of a specific package in the context of the current lockfile? I am hesitant to encourage too many hacks to resolution logic, which is already complicated and difficult to debug.

@brettdh I think I can shed some light because you have most of the pieces. pipenv lock doesn't install anything, and it doesn't claim to. It only generates the lockfile given your host environment, python version, and a provided Pipfile. If you manipulate your environment in some other way or if you use pip directly/manipulate pip settings outside of pipenv / are not using pipenv run or using pip freeze inside a pipenv subshell, it is quite easy for a lockfile to be out of sync from pip freeze. The two aren't really related.

To be clear:

1. Pipfile.lock is a strictly-pinned dependency resolution using the pip-tools resolver based on the user's Pipfile
2. If you want to maintain strict pins of everything while upgrading only one package, I believe you can do this by strictly pinning everything in your Pipfile except for the one thing you want to upgrade (correct me if I'm wrong @vphilippon)

As for your lockfile and pip freeze disagreeing with one another, I'd have to know more information, but I believe we have an open issue regarding our lockfile resolver when using non-system versions of python to resolve.

[k4nar]

@techalchemy : If I have a Pipfile.lock with A, B and C where B is a dependency of A, I would like to be able to update A and B without updating C, or C without updating A and B. Again of course I can pin all my dependencies & their dependencies in my Pipfile in order to do that, but that would be a burden to maintain (like most requirements.txt are).

[brettdh]

I concur with everything @k4nar wrote. Sure, I could even just pin everything in requirements.txt and not use pipenv. The point of pipenv is to have one tool that makes that (and the virtualenv stuff, of course) simpler to manage; i.e. all packages are locked by default to a version that’s known to work, but it should be straightforward to upgrade a select few (without unexpectedly upgrading others). On Thu, Oct 26, 2017 at 4:28 AM Yannick PÉROUX notifications@github.com wrote:

@techalchemy https://github.com/techalchemy : If I have a Pipfile.lock with A, B and C where B is a dependency of A, I would like to be able to update A and B without updating C, or C without updating A and B. Again of course I can pin all my dependencies & their dependencies in my Pipfile in order to do that, but that would be a burden to maintain (like most requirements.txt are).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kennethreitz/pipenv/issues/966#issuecomment-339591307, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFlnqUOEKARiFD8kEk3GVczF3NXBdVOks5swEKcgaJpZM4QEf-- .

[techalchemy]

Hm I see what you guys are saying. The premise of passing a setting to pip is not what I’m worried about, it’s resolving with pip-tools that concerns me. What does this behavior look like right now?

[brettdh]

@techalchemy I mentioned the pip freeze difference as a shorthand for "the package versions that pipenv install installs differ from the package versions that pipenv lock saves to Pipfile.lock."

True, this only happens when I've changed pip's default args via environment variable; I was just pointing out that it was surprising that pipenv delegated to pip for installation but not for version locking; i.e. rather than locking what's installed, it locks what it thinks should be installed, potentially with unrequested upgrades.

Could you clarify your question a bit? I think "resolving with pip-tools" is referring to what pipenv lock is doing, and the reason it's not affected when I set pip defaults? And could you be more specific about what you mean by "this behavior"?

[vphilippon]

@brettdh The locking mechanism include a notion of "dependency resolution" that does not exist in pip. Its handled by pip-tools (or rather, a patched version of it, integrated in a special way by pipenv that bring a few differences with the original tool). In short, the locking mechanism reads the Pipfile and performs a full dependency resolution to select a full set of package that will meet every constraints defined by the required packages and their dependencies.

@techalchemy

[...] it’s resolving with pip-tools that concerns me.

I'm not sure how those --upgrade-strategy would affect pip-tools, because it works on some low-level internals of pip. I have the feeling this would not give the expected result, as these option take into account what's installed, and that's not what's being dealt with in that mechanism. But we have another approach to this in pip-tools that could be done here.

The "original" pip-tools behavior is that it only updates what's is needed in the lockfile (in its context, its the requirements.txt), but this was "lost" in the way the resolver was integrated in pipenv. Let me explain why.

Pointing back to my resume of how pip-tools works: https://github.com/kennethreitz/pipenv/issues/875#issuecomment-337717817 Remember the "select a candidate" part? That's done by querying the Repository object. In pipenv, we directly configure a PyPIRepository for the Resolver, but pip-tools does something else, it uses a LocalRequirementsRepository object, which keeps the existing pins from the previously existing requirements.txt (if found), and "fallbacks" on PyPIRepository.

So in pip-tools, the following happens when selecting a candidate:

1. Query LocalRequirementsRepository for a candidate that match foobar>=1.0,<2.0.
2. Check if an existing pin meets that requirements:
   • If yes, return that pin as the candidate.
   • If not, query the proxied_repository (PyPIRepository) for the candidate.
3. Use the candidate returned

Effectively, it means that existing pins are given a "priority" as candidate to try first.

But in pipenv, currently, it simply:

1. Query PyPIRepository (directly) for a candidate that match foobar>=1.0,<2.0.
2. Use the candidate returned.

So, I think the same behavior for the locking in pipenv could be done by parsing the Pipfile.lock to get the existing pins and use a LocalRequirementsRepository, like pip-tools does in its pip-compile command.

[techalchemy]

@vphilippon do you have a sense of how difficult implementation on that would be?

[vphilippon]

@techalchemy

• Parsing the Pipfile.lock to extract the existing pins: Haven't looked at that. Depends on how things are structured in pipenv. We need a set of InstallRequirements that represents the pins in the Pipfile.lock.
• Using LocalRequirementsRepository: Fairly easy: change our current PyPIRepository for a LocalRequirementsRepository.

---

But, as I'm looking into this, and following @brettdh comments, I realize a few things:

1. The current default pipenv install behavior doesn't match the pipenv lock behavior. Doing pipenv install requests alone won't update requests if a new version comes out (much like straight pip install). However, doing pipenv lock will update the Pipfile.lock with the latest version of requests that matches the Pipfile specifier, and the dependency constraints. There's 2 main way to see this:

   • A) The Pipfile.lock should stay as stable as possible by default, not changing pins unless required, in order to stay like the current environment, and only change in the event that we change the environment.
   • B) The Pipfile.lock should get the newest versions that respect the environment constrains/dependencies in order to freely benefit from the open ranges in the Pipfile and lib dependencies, allowing to continuously acquire new compatible versions in your environment. You can then run pipenv update to benefit from the fresh lock.

   IMHO, I would align the default behavior, which would be to go with A) by default. Because right now, everytime a lock is performed (i.e. after each installation), new versions can come in, which make the lockfile drive the update of the environment, which seems weird. But, this is arguable of course. While in development, I might want to continuously update my requirements to no get stale, like with B), so that should also be easily doable.

2. Even if we use LocalRequirementsRepository to avoid updating correct existing pins, and end up aligning the default behaviors, we then need to address the equivalent of --upgrade and --upgrade-strategy for the locking part. Currently, defining some environment variable (like PIP_UPGRADE and PIP_UPGRADE_STRATEGY) will affect the pipenv install behavior, but will not affect pipenv lock, as it doesn't affect the behavior of pip-tools (I confirmed that, as I was unsure at first). Otherwise, there will be no way to update the environment without either deleting the Pipfile.lock (feels clunky, and "all or nothing") or requiring a newer version (I mean doing an explicit pipenv install requests>2.18.4, which requires you to know that a new version is out, and changes the specifier in the Pipfile itself, increasing the lower bound), which is wrong. As the "original pip-tools" doesn't deffer to pip to deal with this (as it's not related that what is currently installed), it offers an option to specify the dependencies to update in the lockfile, and simply remove the pins for these packages (or all) from the existing_pins list, effectively falling back to querying PyPI. I'm not sure how we can match the notion of "--upgrade-strategy" with this.

---

@techalchemy So while I was saying it was fairly easy to just "align the default behavior", I now realize that this would cause some major issue with being able to update the packages (as in: just fetch the latest version that match my current constraints).

If there's something unclear, ask away, a lot of editing went on when writing this.

(Dependency resolution is not easy. Good and practical dependency resolution is even worst 😄 )

[techalchemy]

@vphilippon that's exactly what I meant. Keeping the things that pip installs in sync with the things that pip-tools resolves is non-trivial unless you drive the process backwards, using the resolved lockfile to do the installation. I'm pretty sure that was why things were designed the way they were.

B) The Pipfile.lock should get the newest versions that respect the environment constrains/dependencies in order to freely benefit from the open ranges in the Pipfile and lib dependencies, allowing to continuously acquire new compatible versions in your environment. You can then run pipenv update to benefit from the fresh lock.

This workflow can possibly work with the current configuration. You can use pipenv lock to generate a lockfile, but pipenv update will reinstall the whole environment. I'm pretty sure we can use one of our various output formats to resolve the dependency graph (we already have a json format as you know) and only reinstall things that don't align to the lockfile. This might be more sensible, but I would be curious about the input of @nateprewitt or @erinxocon before making a decision

[brettdh]

@vphilippon Totally agree that A and B are desirable workflows in different situations. Some of your phrasing around B confused me a bit, though, seeming to say that pipenv lock might result in a lockfile that doesn't actually match the environment - I particularly heard this in that one would need to "run pipenv update to benefit from the fresh lock" - as if the lock is "ahead" of the environment rather than matching it.

Regardless of whether you are in an A workflow or a B workflow, a few things seem constant to me, and I think this squares with what @techalchemy is saying as well:

• The result of pipenv lock should always be a lockfile that matches the environment.
• The result of pipenv install should always be an environment that matches the lockfile.

I'm ignoring implementation details, but that's kind of the baseline behavior I expect from a package manager with a lockfile feature.

Running pipenv update periodically allows you to stay in B mode as long as you want everything to be fresh, and having the ability to pipenv install --upgrade requests would allow specific updates of one package and its dependencies, without affecting packages that don't need to be upgraded unnecessarily.

Am I missing any use cases? I can think of optimizations for B - e.g. a flag or env var that tells it to always update eagerly - but I think that covers the basics. I also know I'm retreading ground you've already covered; it's just helpful for me to make sure I understand what you're talking about. :)

[techalchemy]

Some of your phrasing around B confused me a bit, though, seeming to say that pipenv lock might result in a lockfile that doesn't actually match the environment

@brettdh this is correct -- the pip-tools resolver we use to generate Pipfile.lock doesn't ask the virtualenv for a list of which packages have been installed. Instead, it compiles a list of packages that meet the criteria specified in the list of pins from the Pipfile. Because the resolver itself runs using the system or outer python / pipenv / pip-tools install, we are doing some supreme fuckery to convince it to resolve packages with the same version of python used in the virtualenv. The assumption would be that pip install would resolve things similarly, but that isn't always the case, although even I'm not 100% sure about that. But yes, pipenv lock is not generated based on the virtualenv, it is generated based on the Pipfile. It is a dependency resolution lockfile, not an environment state pin.

[ncoghlan]

As a potential resolution to this: something that pip itself currently supports, but pip-compile doesn't, is the notion of a constraints file.

A constraints file differs from a requirements file, in that it says "If this component is installed, then it must meet this version constraint". However, if a particular package in the constraints file doesn't show up in the dependency tree anywhere, it doesn't get added to the set of packages to be installed.

This is the feature that's currently missing from pipenv, as the desired inputs to the Pipfile.lock generation are:

1. The updated Pipfile contents as a new requirements input file
2. The full set of existing dependencies from Pipfile.lock as a constraints file, excluding the packages specifically named in the current command

Constraints file support at the pip-tools resolver level would then be enough for pipenv to support a mode where attempted implicit upgrades of dependencies would fail as a constraint violation, allowing the user to decide whether or not they wanted to add that package to the set being updated.

[kennethreitz]

currently not supported, thanks for the feedback

[taion]

@kennethreitz

Do you mean:

1. This behavior should be changed, but it's not currently a priority,
2. This behavior should be added as something optional, but it's not currently a priority, or
3. This behavior should not be added?

This is a sufficient inconvenience given the inconsistency with how other similar locking package managers work that it would be good to keep this open as a solicitation for PRs.

If instead it's (3), and this will not be added, then I think a number of us on the issue will need to adjust our plans for our choice of Python package management tools.

[kennethreitz]

I mean that this is currently not supported, and I appreciate the feedback.

[taion]

I understand that it's not supported. Are you also saying that you would not accept PRs either changing this behavior or adding this as an option?

[kennethreitz]

I have no idea.

[brettdh]

@k4nar still interested in doing a PR for this? Specifically, something like pipenv install --only <dep-to-update which prevents unrelated deps from being updated. Since @kennethreitz seems uninterested in discussing further, it seems to me that that's the only way to find out whether that behavior addition/change could be acceptable (and, by extension, whether folks like @taion and I can continue using pipenv).

[k4nar]

I'm interested but I'm not sure to know how would be the best way to implement this. There are a lot of components in action (pip, pip-tools, pipfile, pipenv…) and probably a lot of possible solutions.

[taion]

Per https://github.com/kennethreitz/pipenv/issues/966#issuecomment-339707418, it should be relatively straightforward. That dep resolution logic is largely just from pip-tools. I was planning on submitting a PR, but I can't justify spending the work if we're not willing to talk about how we want the API to look before we spend time writing code.

I'm currently looking at taking an alternative approach – as Pipfile is a standard, interactions with it don't need to go through pipenv, and I'd like to work around some of the other odd semantics here like wiping existing virtualenvs per https://github.com/kennethreitz/pipenv/issues/997.

[rfleschenberg]

Sorry to comment on a closed issue, but I'd like to point out that, to my understanding, using pipenv in my projects currently requires a workflow like this:

pipenv install foo
vim Pipfile.lock  # Manually remove all the unwanted updates
git add && git commit && git push

I find it really annoying having to communicate this to my team members. The alternative seems to be to pin everything to exact versions in Pipfile, but that defeats much of the purpose of using pipenv in the first place.

IIUC, this behavior is the equivalent of apt performing an implicit apt dist-upgrade whenever you run apt install foo.

This is made worse by the fact that pipenv install updates stuff in Pipfile.lock, but does not install the updates into the local virtualenv. If the developer does not carefully examine the diff of Pipfile.lock, they are still using the older versions locally, but once they share the code, all other environments see the surprising updates. People have a tendency to ignore the diff of Pipfile.lock because it's considered an auto-generated file.

I am strongly convinced that "update everything to the latest version allowed by Pipfile" should be an explicitly requested operation that is separate from "install foo".

[kennethreitz]

should be fixed in master

[marius92mc]

The behaviour is still present, I tested it in pipenv 11.8.3, @kennethreitz.

[ncoghlan]

@marius92mc The "fixed in master" comment is referring to the --selective-upgrade and --keep-outdated options added in recent releases: https://docs.pipenv.org/#cmdoption-pipenv-install-keep-outdated

That allows folks that need or want more control over exactly when upgrades happen to opt in to that behaviour, while the default behaviour continues to respect OWASP A9 and push for eager upgrades at every opportunity.

[simonpercivall]

@ncoghlan I think one thing that is needed (easy to ask for, not as easy to do) is an FAQ on how those options behave (at least it's still confusing for me).

For instance: Using --selective-upgrade and --keep-outdated will still cause outdated libraries in the Pipfile.lock to be updated, if they're not directly related to the "selected" package to be updated.

[kennethreitz]

It sounds like there may be implementation bugs, then.

[kennethreitz]

They are intended to leave the pipfile.lock as-is, except for the new change.

[simonpercivall]

Let me know if it's helpful to provide a test Pipfile+.lock.

[kennethreitz]

I think you've provided enough information for us to investigate. I'll try to do that now.

[kennethreitz]

Actually, your pipfile/lock would be great, if it contains outdated results.

[marius92mc]

@ncoghlan, thank you for providing the details. I tried again with your mentioned options and the result seems to be the same, it still updates the other packages as well, changing them in the Pipfile.lock file.

[simonpercivall]

@kennethreitz https://github.com/simonpercivall/pipenv-selective-upgrade-test

[marius92mc]

There are any updates about this issue, @kennethreitz?

[techalchemy]

Sorry for the slow answers on this. We haven’t nailed down the root cause for the regression here yet (I know I personally have been handling a data center migration this weekend so I’ve been kinda slow) but we will get this sorted in the next few days.

Contributions welcome as always!

[wichert]

I think there is a missing use case that can use this same change: when I am developing an application I often need to upgrade a single dependency's version. The steps I would like to follow are:

1. Update the version restriction for the dependency in setup.py
2. Run either pipenv lock --selective-upgrade ; pipenv sync or pipenv install --selective-upgrade "-e ."

[ncoghlan]

@wichert If Pipfile has been edited in a way that increases the minimum required version beyond what's in the current lock file, then --keep-outdated should already cover what you need. --selective-upgrade is for the case where Pipfile hasn't changed, but you want to update to a new pinned version anyway.

[wichert]

@ncoghlan Pipfile has not changed in this scenario, only setup.py by changing the minimum version requirement for a dependency, typically to something more recent and currently in Pipfile.lock.

[techalchemy]

@wichert pipenv doesn't capture changes to your setup.py automatically because it isn't setuptools. You have to run pipenv lock if you want that to happen.

[benkuhn]

What's the current status of this? On March 25th someone said they thought implementation issues would be resolved "in the next couple days", and other bug reports have been closed due to being tracked here; but as of 2018.7.1 I still see the bug reported by Simon Percivall (indirect dependencies are always updated) and that bug hasn't been discussed since the original report. Is the problem still being tracked?

(I'm currently living in a second-tier city in Senegal so my Internet is terrible and it would be a game changer not to blow my data cap on updating indirect dependencies if possible :P )

PS: Thanks for making Pipenv, it's awesome <3

[techalchemy]

Yes for sure. We are rewriting the resolver to support this right now. Whether it lands in this release or next release remains to be seen

[uranusjr]

I’m not that confident with my coding skill to estimate when the resolver would land :p Seriously, this is a completely volunteer project, and we don’t have a deadline mechanism as you would in commercial settings (we don’t even have a boss or a project manager or whatever you have in your company that decides when a thing needs to be done). If you want a thing to be done in a timeframe you desire, you need to do it yourself, or at least provide real motivation for others to do it.

[brettdh]

@uranusjr FWIW, I didn't see any demands for expediency in @benkuhn 's comment above - just a question about where things are at; i.e. what work has been done, so that outside observers can make their own estimates/decisions.

I understand that pipenv is a volunteer project and that non-contributors cannot ask for a thing to be done by a date without signing up to make it happen. I do wonder, whether there is room for more transparency in the project's development process, or if I'm just not looking in the right places. Usually the answer is either "if the issue hasn't been updated, there's been no movement" or "look at this WIP pull request," but this issue in particular seems to have triggered a much larger effort, so the dots can be difficult to connect for those not directly involved.

As always, much thanks to you and everyone who gives their valuable time towards the improvement of pipenv. 👏

[techalchemy]

For sure, this one doesn’t have activity or a work in progress PR because it is a lot more complicated than that. We are talking internally mostly about how we even want to structure this with respect to the larger project, and working iteratively to establish an approach that might even begin to work properly. Once we can sort that out we can build resolution logic.

In the meantime the resolver stack in pipenv is super convoluted and I wouldn’t be comfortable asking people to invest too much effort trying to untangle it for this purpose. Even the simplest use case here will take a significant refactor. We’d be happy to review / discuss any proposed refactor if someone is interested in helping tackle this, but the two things are tightly coupled.

If someone has expertise in dependency resolution and sat solving we would certainly be interested in input but there just isn’t a single concrete idea yet. We’ve been through several iterations that we never planned to carry forward as more than proof of concept. Not all code becomes a PR, and not all code organization decisions happen on the issue tracker. Sometimes we chat synchronously and propose and scrap ideas in real time.

[alecbz]

Something I was going to suggest as an alternative workflow that might address this is making it easy to pin to a specific version in the Pipfile when installing.

I think it's slightly surprising but not completely unreasonable that pipenv interprets foo = "*" to mean "I just need to make sure some version of foo is installed, the user doesn't care which". To that end, having something like pipenv install --pin foo which results in foo = "==1.2.3" instead of foo = "*" in the Pipfile (where 1.2.3 is the current latest version of foo) seems like it might help.

The issue with this though is that the behavior of a lot of packages can change a lot based on their dependencies (e.g., the same version of pylint can do totally different things depending on what version of astroid it's using), and packages don't pin their own deps exactly. So I don't think this actually gets anyone very far. :/