pip --upgrade fails to install some newly added packages

[btbonval]

This problem didn't used to occur. I added the --upgrade flag to pip, which upgrades software if it needs to. Without that, the fabfile would only check to see packages were installed, but if a newer version were listed in the reqs file, nothing would happen.

With --upgrade, missing requirements are installed, but there seems to be an error with one particular package.

It might be best to make a install_reqs fab function (without --upgrade) separate from the update_reqs fab function (with --upgrade), since pip is not clever enough to sort it on its own in rare cases. Seems like a problem that should be reported upstream.

This error is repeatable. Notice the pip command at the bottom.

[127.0.0.1] out:   Running setup.py install for meld3
[127.0.0.1] out:     Traceback (most recent call last):
[127.0.0.1] out:       File "<string>", line 1, in <module>
[127.0.0.1] out:     ImportError: No module named setuptools
[127.0.0.1] out:     Complete output from command /var/www/karmaworld/venv/bin/python -c "import setuptools;__file__='/var/www/karmaworld/venv/build/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-QMQhP5-record/install-record.txt --install-headers /var/www/karmaworld/venv/include/site/python2.7:
[127.0.0.1] out:     Traceback (most recent call last):
[127.0.0.1] out: 
[127.0.0.1] out:   File "<string>", line 1, in <module>
[127.0.0.1] out: 
[127.0.0.1] out: ImportError: No module named setuptools
[127.0.0.1] out: 
[127.0.0.1] out: ----------------------------------------
[127.0.0.1] out: Command /var/www/karmaworld/venv/bin/python -c "import setuptools;__file__='/var/www/karmaworld/venv/build/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-QMQhP5-record/install-record.txt --install-headers /var/www/karmaworld/venv/include/site/python2.7 failed with error code 1 in /var/www/karmaworld/venv/build/meld3
[127.0.0.1] out: Storing complete log in /home/vagrant/.pip/pip.log

Requested: pip install --upgrade -r /var/www/karmaworld/reqs/prod.txt

[btbonval]

Update docs as well.

[btbonval]

Actually, if a pip package is added with this particular bug, an already deployed system will choke.

The proper fix would be to try pip with --upgrade. If that fails, run pip without --upgrade (which will install the bad package) then run it with --upgrade again (to upgrade all the other package requirements to the correct version).

[btbonval]

Tried this. For some reason, when the execution fails in the with settings(warn_only=True) clause, fabric's whole concept of the virtualenv gets borked. Can't seem to find anyone talking about warn_only causing this.

In this example, I test pip prior to running pip --upgrade, and its fine. After pip --upgrade fails on meld3 within a warn_only context, I can't run pip anymore. The error looks very much as though the virtual environment is not being used. This is repeatable, so the virtualenv is not actually modified on disk.

...
> /home/vagrant/karmaworld/fabfile.py(193)update_reqs()
-> with settings(warn_only=True):
(Pdb) virtenv_exec('pip')
[127.0.0.1] run: pip
[127.0.0.1] out: Usage: pip COMMAND [OPTIONS]
[127.0.0.1] out: 
[127.0.0.1] out: pip: error: You must give a command (use "pip help" to see a list of commands)
...
(Pdb) c
...
[127.0.0.1] out:   Running setup.py install for meld3
[127.0.0.1] out:     Traceback (most recent call last):
[127.0.0.1] out:       File "<string>", line 1, in <module>
[127.0.0.1] out:     ImportError: No module named setuptools
...
> /home/vagrant/karmaworld/fabfile.py(197)update_reqs()
-> print "Error running pip install with --upgrade. Trying again without."
(Pdb) virtenv_exec('pip')
[127.0.0.1] run: pip
[127.0.0.1] out: Traceback (most recent call last):
[127.0.0.1] out:   File "/var/www/karmaworld/venv/bin/pip", line 5, in <module>
[127.0.0.1] out:     from pkg_resources import load_entry_point
[127.0.0.1] out: ImportError: No module named pkg_resources

[AndrewMagliozzi]

Is this an urgent bug, Bryan?

On Jan 18, 2014, at 1:06 AM, Bryan Bonvallet notifications@github.com wrote:

Tried this. For some reason, when the execution fails in the with settings(warn_only=True) clause, fabric's whole concept of the virtualenv gets borked. Can't seem to find anyone talking about warn_only causing this.

In this example, I test pip prior to running pip --upgrade, and its fine. After pip --upgrade fails on meld3 within a warn_only context, I can't run pip anymore. The error looks very much as though the virtual environment is not being used. This is repeatable, so the virtualenv is not actually modified on disk.

...

/home/vagrant/karmaworld/fabfile.py(193)update_reqs() -> with settings(warn_only=True): (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Usage: pip COMMAND [OPTIONS] [127.0.0.1] out: [127.0.0.1] out: pip: error: You must give a command (use "pip help" to see a list of commands) ... (Pdb) c ... [127.0.0.1] out: Running setup.py install for meld3 [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "", line 1, in [127.0.0.1] out: ImportError: No module named setuptools ... /home/vagrant/karmaworld/fabfile.py(197)update_reqs() -> print "Error running pip install with --upgrade. Trying again without." (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "/var/www/karmaworld/venv/bin/pip", line 5, in [127.0.0.1] out: from pkg_resources import load_entry_point [127.0.0.1] out: ImportError: No module named pkg_resources — Reply to this email directly or view it on GitHub.

[btbonval]

Nah. It'll fix a 1% of the time possible error. I thought it'd be easy to fix, but when I saw that error, I logged it and moved on. On Jan 18, 2014 8:58 AM, "Andrew Magliozzi" notifications@github.com wrote:

Is this an urgent bug, Bryan?

On Jan 18, 2014, at 1:06 AM, Bryan Bonvallet notifications@github.com wrote:

Tried this. For some reason, when the execution fails in the with settings(warn_only=True) clause, fabric's whole concept of the virtualenv gets borked. Can't seem to find anyone talking about warn_only causing this.

In this example, I test pip prior to running pip --upgrade, and its fine. After pip --upgrade fails on meld3 within a warn_only context, I can't run pip anymore. The error looks very much as though the virtual environment is not being used. This is repeatable, so the virtualenv is not actually modified on disk.

...

/home/vagrant/karmaworld/fabfile.py(193)update_reqs() -> with settings(warn_only=True): (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Usage: pip COMMAND [OPTIONS] [127.0.0.1] out: [127.0.0.1] out: pip: error: You must give a command (use "pip help" to see a list of commands) ... (Pdb) c ... [127.0.0.1] out: Running setup.py install for meld3 [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "", line 1, in [127.0.0.1] out: ImportError: No module named setuptools ... /home/vagrant/karmaworld/fabfile.py(197)update_reqs() -> print "Error running pip install with --upgrade. Trying again without." (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "/var/www/karmaworld/venv/bin/pip", line 5, in

[127.0.0.1] out: from pkg_resources import load_entry_point [127.0.0.1] out: ImportError: No module named pkg_resources — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/282#issuecomment-32682519 .

[AndrewMagliozzi]

cool

On Sat, Jan 18, 2014 at 4:17 PM, Bryan Bonvallet notifications@github.comwrote:

Nah. It'll fix a 1% of the time possible error. I thought it'd be easy to fix, but when I saw that error, I logged it and moved on. On Jan 18, 2014 8:58 AM, "Andrew Magliozzi" notifications@github.com wrote:

Is this an urgent bug, Bryan?

On Jan 18, 2014, at 1:06 AM, Bryan Bonvallet notifications@github.com

wrote:

Tried this. For some reason, when the execution fails in the with settings(warn_only=True) clause, fabric's whole concept of the virtualenv gets borked. Can't seem to find anyone talking about warn_only causing this.

In this example, I test pip prior to running pip --upgrade, and its fine. After pip --upgrade fails on meld3 within a warn_only context, I can't run pip anymore. The error looks very much as though the virtual environment is not being used. This is repeatable, so the virtualenv is not actually modified on disk.

...

/home/vagrant/karmaworld/fabfile.py(193)update_reqs() -> with settings(warn_only=True): (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Usage: pip COMMAND [OPTIONS] [127.0.0.1] out: [127.0.0.1] out: pip: error: You must give a command (use "pip help" to see a list of commands) ... (Pdb) c ... [127.0.0.1] out: Running setup.py install for meld3 [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "", line 1, in [127.0.0.1] out: ImportError: No module named setuptools ... /home/vagrant/karmaworld/fabfile.py(197)update_reqs() -> print "Error running pip install with --upgrade. Trying again without." (Pdb) virtenv_exec('pip') [127.0.0.1] run: pip [127.0.0.1] out: Traceback (most recent call last): [127.0.0.1] out: File "/var/www/karmaworld/venv/bin/pip", line 5, in

[127.0.0.1] out: from pkg_resources import load_entry_point [127.0.0.1] out: ImportError: No module named pkg_resources — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/FinalsClub/karmaworld/issues/282#issuecomment-32682519>

.

— Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/282#issuecomment-32693420 .

[btbonval]

Rare problem which only affected the Vagrant VM and possibly local dev systems.

No vagrant, no dev systems, no problem.