Closed Suor closed 8 years ago
Hi Alexander and thanks for the feedback!
Short answer without details: The scipy
package doesn't actually compile the second time: It is successfully cached and on the second run the cached binary package is correctly re-used. If you are hesitant to accept this explanation I created a gist that demonstrates my statement. As shown in the linked gist the first run takes 14 minutes and the second run takes 2 minutes (the 12 minute difference there is that scipy
doesn't have to be compiled the second time :-). However subsequent runs are slower than they could be and I can explain why.
Explanation of cause of slow down: The scipy
package depends on the numpy
package and includes the numpy
package in its setup.py
script as a setup_requires dependency. What this comes down to is that every time the setup.py
script of scipy
is run, it will download and build the numpy
binary (this is simplifying a bit, the next section contains more details).
General rant about setup_requires "feature" (with more details): To be very blunt the setup_requires
feature of setuptools
is a very tricky feature that is hard to use "correctly" and consequently most people indeed don't use it "correctly", so the effect is quite annoying and easy to demonstrate:
scipy
source distribution archive (the *.tar.gz
file) and unpack it.scipy
source distribution and run an "innocent looking" command like python setup.py --version
and be amazed while the answer to this very simple question takes a couple of minutes because numpy
is being downloaded and compiled before the version number of the scipy
source distribution is printed ಠ_ಠsetup_requires
dependency is already satisfied (you can check with ls -ld numpy-*.egg
), but if you unpack the source distribution to a clean directory and run step 3 again it will again be slow.How this problem can be resolved: I've actually fixed this exact issue before by sending pull requests to Python projects, see e.g. pyca/cryptography#1257. So to bring this issue to some sort of conclusion: It is possible to fix this, but it requires cooperation from the package author(s) because I can't think of any way for pip-accel to fix this issue "from the outside" - the issue can be clearly demonstrated without invoking pip-accel at all :-).
Given my extensive explanation here and the additional information available in pyca/cryptography#1257, do you feel like creating a pull request for the scipy project?
I can do it as well, but 1) I have more than 30 open source projects to maintain and am already slacking on half of them because I just don't have the time and 2) I've never actually used scipy
and have zero experience with it (I actually spent quite a bit of time determining its build dependencies before I was able to reproduce this issue! :-)
A small follow up:
I couldn't believe that this "issue" would have gone unfixed in such a popular / high profile Python package as scipy and it actually looks like the scipy people are aware of the issue and working on it, but between all of the issues, pull requests and commits I still don't see a full solution emerging, maybe I'm confused at this point. The most useful reference I was able to find is scipy/scipy#453. Note that the last comment in that pull request is only two months old and states that the issue hasn't been fully fixed in a released version, if I understand correctly.
Hello, and thank you for this explanation.
I, however, think you can fix it from outside, with some hustle :) Here is what you can do:
bdist_dump
look for ./.eggs
directory and save it along with binary dump,.eggs
to current directory, thus setup_requires
will be satisfied.Alternatively you can keep .eggs
directory for each python with all sorts of stuff and symlink it into current dir before installation.
Sorry for the long delay in replying here, other issues and projects got in the way of finishing this reply and summarizing my thoughts about your proposal to fix this from the outside.
If it turns out that there is a way to:
setup_requires
dependenciesThen I'm all for it and don't mind spending time on implementing this. However I'm pretty sure things are not quite as simple as you explain it in your last comment, and this is the reason why I never seriously tackled this issue inside pip-accel before now:
pip-accel install PKG_NAME
or pip-accel install -r REQ_FILE
. The way this works internally is that pip-accel asks pip to (1) download and (2) unpack the requested Python packages and (3) collect metadata about the packages. The first feedback pip-accel gets from pip after this is when step (3) has already happened. But step (3) is the step which invokes the installation of setup_requires
packages. Catch-22! So properly speaking there is no place for me to put the proposed "Please ensure this .eggs
symbolic link exists" logic.
setup.py egg_info
it does so in the directory that contains the setup.py
script. So during a single run of pip-accel we could be talking about a dozen "working directories" that each get their own setup_requires
packages.python setup.py egg_info
command that triggers the installation of setup_requires
packages. Looking into the pip source code it seems that I would have to monkey patch / override one or two of the below methods:
pip.req.run_egg_info()
pip.req.req_install.run_egg_info()
.eggs
directory which was added in setuptools 7.0. Before that version setuptools would create *.egg
directories in each working directory. Theoretically I could pro-actively link all of these *.egg
directories for each unpacked source distribution, but I'm not going to do this because it's madness. So if I want this to work for all users I'd need to make sure that setuptools 7.0 or higher is installed. The natural way to do that is to just give pip-accel a setuptools >= 0.7
dependency.setup_requires
packages turns out to work, users would still be compiling setup_requires
packages more times than strictly necessary because the setuptools .eggs
cache and the pip-accel binary cache aren't shared, however of course this is still a lot better than recompiling setup requirements on every single run!The best way for me to find out how realistic all of this is would be to (try to) implement the required changes. The issues caused by setup_requires
have been a thorn in my side even before I created pip-accel and since then it hasn't gotten any better, so believe me when I say that I'd love to improve how this works :-).
However there is also issue #57 suggesting to upgrade from pip 6.x to pip 7.x and it seems wise to tackle that upgrade before I introduce yet more monkey patching (as explained above) because every additional pip monkey patch in pip-accel makes it a bit harder for me to upgrade to a new major version of pip.
Would be nice to se it finally :), but no rush.
Hi Alexander,
Sorry things took so long, however thanks for your persistence in fixing this on the side of pip-accel. I just released pip-accel 0.39 which 1) depends on setuptools >= 7.0
and 2) manages the creation of .eggs
symbolic links to avoid recompilation of setup requirements.
I've now tested this with a couple of packages including the SciPy / NumPy combination and it seems to work very well! There is even an automated test to verify the functionality - getting the test to work correctly in all environments was actually a lot more work than the feature itself :-).
Given that I now have a way to manipulate unpacked source distributions before they are processed by pip I'm considering extending this logic to inject allow_hosts
and find_links
options that can keep setuptools (easy_install
) off the internet. Not sure how that would work yet, but I'm thinking about it :-).
Steps to reproduce:
Then destroy virtualenv and repeat, scipy still compiles.