search

paylogic / pip-accel

pip-accel: Accelerator for pip, the Python package manager
https://pypi.python.org/pypi/pip-accel
MIT License
308 stars 35 forks source link

multiple parallels pip-accel can fail #54

Closed oppianmatt closed 9 years ago

oppianmatt commented 9 years ago

even with a virtualenv.

We got parallel branches being build/tested with jenkins and it uses pip-accel to install packages into a virtual env. But it seems sometimes we get an error about missing files. My guess is that the other pip-accel deleted it. 

015-04-14 11:31:43 stagingpitchup.dh.bytemark.co.uk pip_accel.bdist[4504] INFO Deleting old functools32 (3.2.3.post1) binary (source is newer) ..
2015-04-14 11:31:44 stagingpitchup.dh.bytemark.co.uk pip_accel.cli[4504] ERROR Caught unhandled exception!
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/cli.py", line 55, in main
    accelerator.install_from_arguments(arguments)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 234, in install_from_arguments
    return self.install_requirements(requirements, **kw)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 520, in install_requirements
    self.bdists.install_binary_dist(binary_distribution, **kw)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 329, in install_binary_dist
    for member, from_handle in members:
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 76, in get_binary_dist
    os.unlink(cache_file)
OSError: [Errno 2] No such file or directory: '/var/lib/jenkins/.pip-accel/binaries/v7/functools32:3.2.3.post1:CPython-2.7.tar.gz'
xolox commented 9 years ago

Hi Matthew!

Thanks for the clear bug report and sorry for breaking your builds. I added the os.unlink() call without properly thinking through the repercussions of parallel builds. I just released pip-accel 0.28.2 which removes the os.unlink() call, leaving it to the existing logic in the local cache backend to atomically move the cached binary distribution into place.

Can you try out pip-accel 0.28.2 and let me know whether it resolves the problem you reported?

xolox commented 9 years ago

I'm confident that the problem you reported here is now resolved so I'm going to close this issue. If you believe the problem you reported is not fixed in pip-accel 0.28.2 and newer versions feel free to reopen this issue or open a new one. Thanks for the feedback!