search

simion / pip-upgrader

An interactive pip requirements upgrader. It also updates the version in your requirements.txt file.
Apache License 2.0
512 stars 35 forks source link

Crashes if pip-upgrader exists in requirements.txt #4

Open eddie-dunn opened 7 years ago

eddie-dunn commented 7 years ago 
% pip-upgrader
<snip>
Traceback (most recent call last):
  File "/home/username/.virtualenvs/venv/bin/pip-upgrade", line 11, in <module>
    sys.exit(main())
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/cli.py", line 70, in main
    packages, options.get('--use-default-index')).detect_available_upgrades(options)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 126, in detect_available_upgrades
    package_status, reason = self._fetch_index_package_info(package_name, current_version)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 167, in _fetch_index_package_info
    return self._parse_simple_html_package_info(package_name, current_version, response)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 228, in _parse_simple_html_package_info
    latest_version = max([vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease])
ValueError: max() arg is an empty sequence
% pip-upgrade --version
1.4.0
% python --version
Python 3.5.3

You might not want to support running pip-upgrader on itself, but I don't think the app should crash because someone has put it in requirements.txt.

simion commented 7 years ago

I have it in the requirements and it works. Please paste yoyr requirements.txt here or email it to me at simion.agv@gmail.com and i will check which package crashes. Also, this is a good opportunity to catch all upgrade errors for a single package, print them to console and continue upgrading with the rest of the packages. This way, if any other strange package is added, you won't be blocked.

eddie-dunn commented 7 years ago

Ah, the problem was that I generated the file with pip freeze > requirements.txt, which locked pip-upgrader to a specific version.

requirements.txt:

...
pip-upgrader==1.4.0
...

crashes.

requirements.txt:

...
pip-upgrader
...

works.

simion commented 7 years ago

Still wierd, it shouldn't crash.

simion commented 7 years ago

I'm going to release a fix for it anyway.

eddie-dunn commented 7 years ago

Could you reproduce the bug? For some reason

[vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease]

produces an empty list. Adding a guard clause before running max() on it should work, but perhaps the underlying issue is triggered by the filtering done on the previous lines.

simion commented 7 years ago

Released version v1.4.1 with some error catching, please check it out.

simion commented 7 years ago

I cannot reproduce it 😞

eddie-dunn commented 7 years ago

I can't reproduce it on my laptop at home either :/

I.e., both versions worked fine with pip-upgrader==1.4.0 in requirements.txt. I will see if I can find the root cause once I get back to work on Monday.

simion commented 7 years ago

Cool, thanks. If it breaks at work, it should print the problematic package and continue. For this, you'll have to upgrade pip-upgrader first. pip-upgrade -p pip-upgrader

eddie-dunn commented 7 years ago

The problematic package is pip-upgrade, in my venv at work. I've done some digging and it seems the line

versions_match = re.findall(pattern, response.content.decode('utf-8'), flags=re.IGNORECASE)

produces an empty list for the following response.content string

<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href="../../packages/e7/1c/5621b74ed130a6c6331865d4e05c6be69e3a703dd8a2a6e672af15156120/pip_upgrader-0.1.0-py2.py3-none-any.whl#md5=13cd80dd4e0abfbf711626cdef323b08" rel="internal" >pip_upgrader-0.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/7f/95/2a8f3fbad56f0c044fab9d196b4d53906c892239223d0c0d271561993305/pip_upgrader-0.1.0.tar.gz#md5=9c9303e83734d99dc692a496217296a8" rel="internal" >pip_upgrader-0.1.0.tar.gz</a><br/>
<a href="../../packages/8b/29/2b7839745e5c977fe3644cb2ac0c98e1bec546677587cfbaa96951dd1eb7/pip_upgrader-0.1.1-py2.py3-none-any.whl#md5=b97e169e6b9c03b222837a98655ec9f2" rel="internal" >pip_upgrader-0.1.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/30/a7/88ca8de3f2cbb2a3f53d6bb7d5c83d470eb205c0900f104c420be0911e0f/pip_upgrader-0.1.1.tar.gz#md5=e2b386ea95010de240625c4b63a94ae5" rel="internal" >pip_upgrader-0.1.1.tar.gz</a><br/>
<a href="../../packages/18/86/c34e458362282dfcd704c70d54f4432dc84779b706c29a4413128aa8736b/pip_upgrader-1.0.0-py2.py3-none-any.whl#md5=cd43d3fe7e5ef64e0a084e086360b87f" rel="internal" >pip_upgrader-1.0.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a8/4c/1b0fd5da99ff288f9954c8abc5ea0b08496bbdf361619fd15ae6cdd004bd/pip_upgrader-1.0.0.tar.gz#md5=75b70da482de63fc6dd0fe9e5fabc3f8" rel="internal" >pip_upgrader-1.0.0.tar.gz</a><br/>
<a href="../../packages/4b/a2/bd86365aff1f1ab5c1c47bc5f0311b3a94d151395314cc192acb80918a82/pip_upgrader-1.0.1-py2.py3-none-any.whl#md5=ce7eec58c7cde9f9c708c74ebd77447a" rel="internal" >pip_upgrader-1.0.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/e9/8cc706ef5f97ebbaed3074f1eb87bd453ec6e03953135f7d8887243ee711/pip_upgrader-1.0.1.tar.gz#md5=fb3ee8cb0c214e19add0e017b08864d0" rel="internal" >pip_upgrader-1.0.1.tar.gz</a><br/>
<a href="../../packages/d5/54/73e845a3fd5eaa886bb514420e49c819e06b6dde965fd44640c48f5fe014/pip_upgrader-1.1.0-py2.py3-none-any.whl#md5=6c954879e2bc68bcd3291fbef0041f7c" rel="internal" >pip_upgrader-1.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/33/bd/f89d891fdb4cf0bd1ea9403cfd50735f8d51d89d4fbbea122fa8b7771990/pip_upgrader-1.1.0.tar.gz#md5=2fcb2ffa4b681ce99e4779ae5abe0d20" rel="internal" >pip_upgrader-1.1.0.tar.gz</a><br/>
<a href="../../packages/69/4f/b0fa5f234b7be7a9bfb081713101e50db8f9353a7b6aec81601f65751b0d/pip_upgrader-1.2.0-py2.py3-none-any.whl#md5=27a85b7027de08fbdf19dfc8f97a45f3" rel="internal" >pip_upgrader-1.2.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/82/03/e8992bb500bdd9e228b898315fde69b6e93ef6182a54b89deb5a908d6985/pip_upgrader-1.2.0.tar.gz#md5=e7da3594fc6f79a65ea09186a71c411e" rel="internal" >pip_upgrader-1.2.0.tar.gz</a><br/>
<a href="../../packages/87/b7/47293388a134d651bfb00983a7ac6bd57093f6513dc77a1d0b4abb9761cf/pip_upgrader-1.3.0-py2.py3-none-any.whl#md5=27fb3f805ec291336379817192c673e0" rel="internal" >pip_upgrader-1.3.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/5c/ef/e00b75c4d933ddfd80319f788f3a266a96f474983e237ae150cd234fcdc0/pip_upgrader-1.3.0.tar.gz#md5=c1c0c8388a3419859fb7b152adfd005b" rel="internal" >pip_upgrader-1.3.0.tar.gz</a><br/>
<a href="../../packages/c4/b2/f4b3cf8b93ea34972a181855026021ae7ab41985280c8873baa0cc31965f/pip_upgrader-1.3.3-py2.py3-none-any.whl#md5=6a72e786c29e782b64b6d3b90554df4e" rel="internal" >pip_upgrader-1.3.3-py2.py3-none-any.whl</a><br/>
<a href="../../packages/90/b9/32bc955ab0b9ef7287db5d3422baac0019ec1a3023dd84506c9aa21524a1/pip_upgrader-1.3.3.tar.gz#md5=7f31f9f1184eceac90d65d3134d877cd" rel="internal" >pip_upgrader-1.3.3.tar.gz</a><br/>
<a href="../../packages/62/f7/4d9609f1b4b98d7f4e7edaabc57e0d812985c198f8c90c45c46197871067/pip_upgrader-1.3.5-py2.py3-none-any.whl#md5=ab83e627298a0a16f4ade8b0aa4024c2" rel="internal" >pip_upgrader-1.3.5-py2.py3-none-any.whl</a><br/>
<a href="../../packages/24/f2/4f6262ccd26692cda30e41aa40f44d338fd7f6744a699260ae62692bd99f/pip_upgrader-1.3.5.tar.gz#md5=f1c329fdcc5f3129dd4b2306c6615ffd" rel="internal" >pip_upgrader-1.3.5.tar.gz</a><br/>
<a href="../../packages/90/41/55fd034f13780c9ced6220e23a8b286b8577e77c323cfc7502c39d210525/pip_upgrader-1.4.0-py2.py3-none-any.whl#md5=ab7d4df56b1dc98dd00926793f227511" rel="internal" >pip_upgrader-1.4.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/aa/bbec7df19a29c2d3ded829c136116e50d0c2f86287b24fb4162aa9521db8/pip_upgrader-1.4.0.tar.gz#md5=1b401ef39ba8df949427fc422518bd55" rel="internal" >pip_upgrader-1.4.0.tar.gz</a><br/>
<a href="../../packages/1f/44/9135c9e4434ffd057e4b5fbed7af57dafd2dcbcc801322f50a2e85d83fe5/pip_upgrader-1.4.1-py2.py3-none-any.whl#md5=ecc1c675db23044b9ba6c9891ed30ad3" rel="internal" >pip_upgrader-1.4.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/62/aa/024a4613dca987ca5b2fc00b99343893ccb40e07478945162b4992b81917/pip_upgrader-1.4.1.tar.gz#md5=71e46108739d6e7a15d91844707e4edf" rel="internal" >pip_upgrader-1.4.1.tar.gz</a><br/>
<a href="../../packages/8d/dc/db9b7612d7df5805c440a5a01c66d902e3f8e0f4ed884d2da0b46099a10e/pip_upgrader-1.4.dev1-py2.py3-none-any.whl#md5=63df490e7dd62c6fb1e2257c70519832" rel="internal" >pip_upgrader-1.4.dev1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/0b/84/daab498858f39b0ddbde46c1744137c53c4b78ccc0a52ad0d091acd096c2/pip_upgrader-1.4.dev1.tar.gz#md5=4c8a906b77315fda56b42cd2effdc5b7" rel="internal" >pip_upgrader-1.4.dev1.tar.gz</a><br/>
</body></html>

I'm no regex guru, but I think the problem is that the html blob only refers to pip_upgrader, when the package_name variable is pip-upgrader. I.e., one has a dash, the other an underscore.

Here's my requirement.txt

apipkg==1.4
astroid==1.5.3
certifi==2017.4.17
chardet==3.0.4
colorclass==2.2.0
coverage==4.4.1
decorator==4.0.11
docopt==0.6.2
execnet==1.4.1
idna==2.5
isort==4.2.15
lazy-object-proxy==1.3.1
mccabe==0.6.1
packaging==16.8
pbr==3.0.1
pep8==1.7.0
pika==0.10.0
pip-upgrader==1.4.1
py==1.4.34
pygerrit2==2.0.2
pylint==1.7.1
pylint-mccabe==0.1.3
pyparsing==2.2.0
pytest==3.1.2
pytest-cache==1.0
pytest-cov==2.5.1
pytest-datadir-ng==1.1.0
pytest-mccabe==0.1
pytest-pep8==1.0.6
pytest-pylint==0.7.1
requests==2.18.1
requests-mock==1.3.0
retry==0.9.2
six==1.10.0
terminaltables==3.1.0
urllib3==1.21.1
wrapt==1.10.10

I can reproduce it by running the following commands

$ mkvirtualenv foo
<snip>
$ pip install -r requirements.txt  # the requirements.txt from above
<snip>
$ pip-upgrade
Found valid requirements file(s): 
requirements.txt
Setting API url to https://artifacts.domain.com/artifactory/api/pypi/all-pypi/simple/{package} as found in /home/username/.config/pip/pip.conf. Use --default-index-url to use pypi default index
1/37: apipkg ... up to date: 1.4
2/37: astroid ... up to date: 1.5.3
3/37: certifi ... up to date: 2017.4.17
4/37: chardet ... up to date: 3.0.4
5/37: colorclass ... up to date: 2.2.0
6/37: coverage ... up to date: 4.4.1
7/37: decorator ... up to date: 4.0.11
8/37: docopt ... up to date: 0.6.2
9/37: execnet ... up to date: 1.4.1
10/37: idna ... up to date: 2.5
11/37: isort ... up to date: 4.2.15
12/37: lazy-object-proxy ... up to date: 1.3.1
13/37: mccabe ... up to date: 0.6.1
14/37: packaging ... up to date: 16.8
15/37: pbr ... up to date: 3.0.1
16/37: pep8 ... up to date: 1.7.0
17/37: pika ... up to date: 0.10.0
error while parsing version
19/37: py ... up to date: 1.4.34
20/37: pygerrit2 ... up to date: 2.0.2
21/37: pylint ... up to date: 1.7.1
22/37: pylint-mccabe ... up to date: 0.1.3
23/37: pyparsing ... up to date: 2.2.0
24/37: pytest ... up to date: 3.1.2
25/37: pytest-cache ... up to date: 1.0
26/37: pytest-cov ... up to date: 2.5.1
27/37: pytest-datadir-ng ... up to date: 1.1.0
28/37: pytest-mccabe ... up to date: 0.1
29/37: pytest-pep8 ... up to date: 1.0.6
30/37: pytest-pylint ... up to date: 0.7.1
31/37: requests ... up to date: 2.18.1
32/37: requests-mock ... up to date: 1.3.0
33/37: retry ... up to date: 0.9.2
34/37: six ... up to date: 1.10.0
35/37: terminaltables ... up to date: 3.1.0
36/37: urllib3 ... up to date: 1.21.1
37/37: wrapt ... up to date: 1.10.10
All packages are up-to-date.

Upgrade interrupted.
$ python -V
Python 3.5.3
$ pip-upgrade --version
1.4.1
eddie-dunn commented 7 years ago

Why this doesn't happen on my laptop, or for you at all, I cannot say.

simion commented 7 years ago

I think it's because at work you have a custom package index (artifact) configured in ~/.config/pip/pip.conf.

After upgrading pip-upgrade (use --default-index-url if you still have errors), does it work? It should print the error in console, if any.

eddie-dunn commented 7 years ago

Yes, you are correct; that worked with no issue.

At work we should generally go through the custom index (which should pull requested packages from PyPI if they don't exist). In other words, do you have any idea why the custom index might cause the issue?

If not, that's OK. The workaround is simple enough that I can use it instead. But a fix would be nice, because unless it's some kind of misconfiguration of the index on our side, others might encounter the problem as well.

simion commented 7 years ago

You're right, right now the upgrader does not fallback to pypi when a package is not found.

To be fixed in next release 😉

simion commented 7 years ago

The solution for this issue is not complicated, but I need to install artifactory to test it properly :)

eddie-dunn commented 7 years ago

Do you know what Artifactory is? It's a commercial paid for product. I'm not sure you need Artifactory to test the issue, but there's a free trial here: https://www.jfrog.com/artifactory/free-trial/

Maybe a regular private package index would be enough to test it, without having to deal with Artifactory.