Closed jquast closed 9 years ago
also, this is on OSX
(xxx)$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.10
BuildVersion: 14A389
Thanks for the bug report and sorry for not following up on this earlier. I did investigate a bit but haven't found anything conclusive. I suspect that this issue is the first time that I see that a user of pip-accel runs into "cache invalidation" issues. If that doesn't make sense to you, consider the following:
*.so
files*.so
files your system needs to be in a state that is ABI compatible with your system at the time that the *.so
files were compiled (this is an oversimplification but it explains the basic idea)To actually verify that this is what happened in your case I need to be able to reproduce it. My wife is running Mac OS X as well so with some luck I'll be able to reproduce this issue on her system. To be continued :-)
Also reproduces on "Mavericks",
ProductName: Mac OS X
ProductVersion: 10.9.5
BuildVersion: 13F34
e-mailed you an offer to provide ssh access to such machine.
Problem analysis
The SSH access helped to quickly narrow down the cause of this problem, so thanks for that :-). It's not at all related to cached binary distributions becoming invalid, instead it has to do with the (IMHO) strange behavior of the setup.py
script of IPython:
setup.py
script of IPython reports a set of requirements in the install_requires
keyword to the setup()
function. Based on this list of requirements pip extends the requirement set and downloads missing source distributions. This process continues until all dependencies are satisfied.python setup.py install
to install packages while pip-accel instead runs python setup.py bdist_dumb
(or python setup.py bdist
) and caches the resulting binary distribution. This is what makes pip-accel faster. When the setup.py
script of IPython sees bdist*
in sys.argv
(the command line arguments) it unconditionally adds gnureadline
to install_requires
. Because pip never runs any bdist*
commands it never sees this dependency, so the dependency is not part of the dependency set. But it is written to the metadata included with the binary distribution, so the first time IPython is used after pip-accel installs it, setuptools notices the missing requirement and complains with:Traceback (most recent call last):
File "/tmp/issue-34-env/bin/ipython", line 5, in <module>
from pkg_resources import load_entry_point
File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
working_set = WorkingSet._build_master()
File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 449, in _build_master
ws.require(__requires__)
File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 742, in require
needed = self.resolve(parse_requirements(requirements))
File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gnureadline
Coming up with a solution
It might be possible for pip-accel to work around this issue but it would be a nasty hack, not an elegant and general solution. I think our best option is to convince the IPython developers to change their setup.py
script. I just created a pull request to accomplish this, see ipython/ipython#7047.
@jquast: As a short term workaround you can add gnureadline
to the pip command line or requirements file. It's a bit pragmatic but it will solve the immediate problem until ipython/ipython#7047 is merged (assuming the IPython developers are willing to merge it :-).
@jquast: My pull request ipython/ipython#7047 has been merged so the changes will eventually find their way to PyPI and until that time there is the (short term) workaround of adding gnureadline
to the pip command line or requirements file. I believe this resolves the issue you reported.
Great job, I'm satisfied. Thank you @xolox
preparing environment for pip-accel
installing ipython==2.3.0 with pip-accel (fails)
installing ipython==2.3.0 with pip (succeeds)