Add "upgrade" and "upgrade-all" commands

vbabiy commented 13 years ago

(Updated to reflect reality in April 2020)

This issue was initially scoped for changing the behavior of pip install --upgrade, by adding a new upgrade command with a different behavior when upgrading packages. The older recursive "eager" default caused grief for many (#304) However, after a lot of discussion on this, the approach taken was to add the --upgrade-strategy flag. The flag initially had a default of "eager" which was changed to a better default in #4500.

The tracking issue for "upgrade all packages" functionality is #4551.

vbabiy commented 13 years ago

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more about "upgrade-all"; for one thing, I presume it would only upgrade packages inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to upgrade global packages). That would be a reason to move UninstallPathSet.can_uninstall() to a more generically-named function (or method of InstallRequirement), so it provides generic "can I touch this?" decisions.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

For the record, I think that seems like good idea, given the ability to uninstall before upgrading. Although I'd prefer an --all option for upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have globally anyway.

Original Comment By: Jannis Leidel

vbabiy commented 13 years ago

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one prerequesite is that it lives in /lib/pythonX.Y/site-packages (simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent environment into a virtualenv environment) then probably packages in that parent environment shouldn't be upgraded, but it's not 100% clear that is what the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite reasonably offered to remove the .egg-link and update easy-install.pth. But it couldn't have reasonably upgraded the package, so can_uninstall is somewhat different from can_upgrade.

Original Comment By: Ian Bicking

vbabiy commented 13 years ago

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade imported global packages; but (along with editables) it might be worth a "not upgrading this" warning to the console.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

+1 for this bug

Original Comment By: smyrman

vbabiy commented 13 years ago

Issue #167 was marked as a duplicate of this issue.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages fails the whole process will fail (it's similar to port upgrade outdated in MacPorts).

Original Comment By: Tomasz Elendt

vbabiy commented 13 years ago

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no one provides simple upgrade + upgrade --all command?

Original Comment By: Anonymous

vbabiy commented 13 years ago

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all' option be added to pip?

Running pip upgrade should only affect the environment it is running in. If run from a virtualenv then only packages local to that env will be upgraded; same for non-virtualenv's

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Kelsey: from my reading of the above discussion, I don't see any real opposition to it. I think it's a fine addition to make. The main edge case is editable VCS installs - as Ian suggested, I think an "upgrade" command shouldn't touch those. Defining what "upgrade" means in the context of all the editable possibilities (including local repos installed editable that have no origin) would be next to impossible, I think, and even if some halfway-working definition could be put together, it would only increase the maintenance burden of the already-fragile VCS backends. But for non-editables -- go for it!

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

Carl: Cool, I will get started and update this ticket with the results.

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

While working on the upgrade command the following questions came up:

What methods should pip upgrade support to specify which packages to upgrade? Should we support a requirements file?
How should pip upgrade handle packages that are not already installed? Should we install missing packages?

pip upgrade use cases and how to handle them:

# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade specific non-editable packages to a new or older version; or you can upgrade all non-editable packages to a newer version.

Thoughts?

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as it works now. That implies that yes, it installs requested packages if they aren't installed, and yes, it accepts requirements files with -r. His should be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently installed upgradable packages; then it should just pass that list to install --upgrade, as above.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

Carl, thanks for the reply. I have pretty much taken the path you have described. Later today I should be able to post some example runs.

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Got most of the code done, just a little polish left before I post the code for review.

TODO:

Tests
Filter requirements file; add non-editables to list of packages to upgrade.

Running pip using the upgrade command

# pip upgrade --all

All packages up-to-date

# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date

# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...

# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date

# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...

# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Last set of questions (I hope):

Should pip upgrade parse the requirements file and filter out editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the indexes for a later version. In order to do this I would need to gather the package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

Should pip upgrade search the indexes for a later version to install? (This is what I am doing now). If not how should the upgrade command determine if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

The package is not installed
An upgrade is available (later version from the indexes), and no explicit version was requested
The requested version is different from the installed one (Version miss match).
I am deferring the requirements file to the install command until I filter out the non-editable requirements.

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Carl after re-reading your post, it seems I am doing more than what is required. I will upload my branch so you can take a look. I may have went overboard by checking PyPi for a later version.

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Carl after re-reading your post, it seems I am doing more than what is required. I will upload my branch so you can take a look. I may have went overboard by checking PyPi for a later version.

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Yeah, it sounds like you're doing more than should be needed. Pip install --upgrade does everything you're discussing already (checking for newer versions, etc); all "pip upgrade" should be is like a two-liner passing everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get the full list of installed upgradeable packages in the environment.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

Yeah, I knew it. Well, I did learn alot. Even though not required for this task, I do like the ability to show whats installed and available during the upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

I have stripped down the upgrade command per Carl's suggestions (I went to far in the first place). Not sure I like the results, but it does mirror install --upgrade in functionality.

It seems pip tries to download and re-install the package even when the package is already installed and up-to-date. Even worse with upgrade --all, pip re-installs everything including pip itself. Is this how pip upgrade should work? If so then I am almost done :)

Running pip upgrade command

# pip upgrade Mako

Downloading/unpacking Mako

  Running setup.py egg_info for package Mako

    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe

Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755

    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe

    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you identified #13, that it re-downloads and re-installs even packages that are already up to date. The solution there isn't to do something different with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself too, if there's an upgrade available. (Personally I will never use upgrade --all, so I don't know what behavior people who will use it would want from it here). Obviously again it'd be better-behaved if #13 were fixed.

Original Comment By: Carl Meyer

vbabiy commented 13 years ago

Thanks Carl, I will wrap this up and start looking at #13

Original Comment By: Kelsey Hightower

vbabiy commented 13 years ago

If anyone has time please review my upgrade-command branch. In the meanwhile I'll work on unittests that try not to duplicate the existing ones for the install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py

Original Comment By: Kelsey Hightower

jedie commented 13 years ago

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

https://github.com/pypa/pip/pull/313 https://github.com/jedie/pip/commit/7a31d70dabe0e809184fe1b5280c5a7ccf420dd5

nisavid commented 12 years ago

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

gimmi commented 12 years ago

+1

piotr-dobrogost commented 12 years ago

@kelseyhightower Any progress?

fradeve commented 12 years ago

+1

januz commented 12 years ago

+1!

sebdah commented 12 years ago

+1

nicolargo commented 12 years ago

+1

jezdez commented 12 years ago

Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.

Instead I'd be thrilled to see a comment "patch done!" ;)

gotlium commented 11 years ago

+1

pradyunsg commented 11 years ago

Any Updates? Is there any plan to add this, it's 3 years old now..

thedrow commented 11 years ago

@pradyun See https://github.com/pypa/pip/issues/59#issuecomment-9418336

robsonpeixoto commented 11 years ago

+1

kelseyhightower commented 11 years ago

I thought this was merged already. I can dust off my python skill and give it another go.

eproxus commented 11 years ago

Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...

riyadparvez commented 11 years ago

+1

chriskrycho commented 10 years ago

What is the status of this issue?

eduardocereto commented 10 years ago

It is blocked by #988

ariccio commented 10 years ago

If it's really important for you, there are workarounds for upgrading all packages; I threw together a script to do so in parallel(https://github.com/ariccio/update-pip-packages), and there are many other implementations elsewhere on the internet.

rgommers commented 10 years ago

There's two parts to this issue. upgrade-all may be blocked by gh-988, but I don't see how upgrade is blocked. pip upgrade can be a simple alias for pip install -U --no-deps. This would resolve one of the main problems of using install_requires in setup.py files. Can't this be done sometime soon?

qwcode commented 10 years ago

pip upgrade can be a simple alias for pip install -U --no-deps

from the description:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

that is not pip install -U --no-deps

“non-recursive” in this context does not simply mean –no-deps. A non-recursive upgrade will upgrade dependencies, but only if needed to fulfill parent requirements.

rgommers commented 10 years ago

@qwcode thanks for the clarification. Then that's not what I care about. Why would you call this "non-recursive", it's still recursive but just a bit smarter/different?

I was under the impression from the discussion in gh-571 that really non-recursive was the desired default. That would certainly make sense, and prevent having to always write code like https://github.com/scipy/scipy/commit/8e7ee0c4b3c16.

qwcode commented 10 years ago

in #571, "non-recursive" is not --no-deps it's the "smarter/different" recursive as you say. notice the test_upgrade_with_needed_recursive_upgrades() test from that PR.

without getting stuck on terms, there's 3 things:

the "smarter/different upgrade", i.e. the kind that occurs in OS packaging that can also imply upgrading dependencies (but only if they actually need upgrading).
what pip does now, which is to upgrade all dependencies, regardless of need or conflict resolution.
--no-deps

some possible ways to distinguish #1 and #2

"non-recursive" vs "recursive"
"normal" vs "recursive"
"only if needed recursive" vs "do it regardless recursive"

I'm open to using the best terms... just not sure what those are.

qwcode commented 10 years ago

I think I'm liking this "only-if-needed recursive" phrase. maybe i should use that here in the docs:

rgommers commented 10 years ago

I like it too. If you'd describe the three options all together as

a. non-recursive
b. only if needed recursive
c. recursive (or "do it regardless recursive")

that would be clear.

Then you want to pick good defaults. For both upgrade and install -U, either (a) or (b) could make sense. I strongly prefer (a), but (b) makes sense too given that that's what OS packaging does.

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U.

rgommers commented 10 years ago

To clarify why I strongly prefer (a): an unsuspecting user wanting (b) and getting (a) will have to read a message saying "non-recursive upgrade can't satisfy all dependencies, please use only-if-needed recursive", which is not that big a deal. If the default would be (b), that unsuspecting user may end up with an upgrade or even a broken install of a package he really didn't want to be touching. That can take hours or even days to recover from.

pypa / pip