search

paylogic / pip-accel

pip-accel: Accelerator for pip, the Python package manager
https://pypi.python.org/pypi/pip-accel
MIT License
308 stars 35 forks source link

Added readonly flag for s3 cache #43

Closed lgtml closed 9 years ago

lgtml commented 9 years ago

Purpose

The purpose of this is to allow CI to keep the s3 cache current and developers will be able to prime their cache from the s3 source.

Tests

Could not see any existing tests for the Cache s3 cache. Here are my manual tests.

Intial with PIP_ACCEL_S3_READONLY=1

2014-11-20 21:34:27 leigh-XPS13 pip_accel[11941] INFO Unpacking source distribution(s) ..
2014-11-20 21:34:27 leigh-XPS13 pip_accel[11941] INFO Executing command: pip install --no-install --no-index --find-links=file:///home/leigh/.pip-accel/sources --build-directory=/tmp/tmpN8Oqrg scriptine
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking scriptine
  Running setup.py egg_info for package scriptine

    warning: no previously-included files matching '*.pyc' found under directory 'test'
Successfully downloaded scriptine
2014-11-20 21:34:27 leigh-XPS13 pip_accel[11941] INFO Finished unpacking 1 source distribution in 0.21 seconds.
2014-11-20 21:34:27 leigh-XPS13 pip_accel[11941] INFO Installing from binary distributions ..
2014-11-20 21:34:28 leigh-XPS13 pip_accel.caches.s3[11941] INFO Checking if distribution archive is available in S3 bucket: Linux-x86_643/v7/scriptine:0.2.0:CPython-2.7.tar.gz
2014-11-20 21:34:28 leigh-XPS13 pip_accel.bdist[11941] INFO Building scriptine (0.2.0) binary distribution ..
2014-11-20 21:34:29 leigh-XPS13 pip_accel.bdist[11941] INFO Finished building scriptine (0.2.0) in 0.41 seconds.
2014-11-20 21:34:29 leigh-XPS13 pip_accel.caches.s3[11941] INFO Skipping S3 Upload, S3 Readonly enabled
2014-11-20 21:34:29 leigh-XPS13 pip_accel[11941] INFO Finished installing 1 requirement in 1.29 second.

Second Clean Run without PIP_ACCEL_S3_READONLY set

2014-11-20 21:34:43 leigh-XPS13 pip_accel[11974] INFO Unpacking source distribution(s) ..
2014-11-20 21:34:43 leigh-XPS13 pip_accel[11974] INFO Executing command: pip install --no-install --no-index --find-links=file:///home/leigh/.pip-accel/sources --build-directory=/tmp/tmpc66UES scriptine
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking scriptine
  Running setup.py egg_info for package scriptine

    warning: no previously-included files matching '*.pyc' found under directory 'test'
Successfully downloaded scriptine
2014-11-20 21:34:43 leigh-XPS13 pip_accel[11974] INFO Finished unpacking 1 source distribution in 0.19 seconds.
2014-11-20 21:34:43 leigh-XPS13 pip_accel[11974] INFO Installing from binary distributions ..
2014-11-20 21:34:43 leigh-XPS13 pip_accel.caches.s3[11974] INFO Checking if distribution archive is available in S3 bucket: Linux-x86_643/v7/scriptine:0.2.0:CPython-2.7.tar.gz
2014-11-20 21:34:44 leigh-XPS13 pip_accel.bdist[11974] INFO Building scriptine (0.2.0) binary distribution ..
2014-11-20 21:34:44 leigh-XPS13 pip_accel.bdist[11974] INFO Finished building scriptine (0.2.0) in 0.41 seconds.
2014-11-20 21:34:44 leigh-XPS13 pip_accel.caches.s3[11974] INFO Uploading distribution archive to S3 bucket: Linux-x86_643/v7/scriptine:0.2.0:CPython-2.7.tar.gz
2014-11-20 21:34:44 leigh-XPS13 pip_accel.caches.s3[11974] INFO Finished uploading distribution archive to S3 bucket in 0.3 seconds.
2014-11-20 21:34:44 leigh-XPS13 pip_accel[11974] INFO Finished installing 1 requirement in 1.68 second.

Third Clean run with PIP_ACCEL_S3_READONLY set and bucket warmed

2014-11-20 21:34:58 leigh-XPS13 pip_accel[12005] INFO Unpacking source distribution(s) ..
2014-11-20 21:34:58 leigh-XPS13 pip_accel[12005] INFO Executing command: pip install --no-install --no-index --find-links=file:///home/leigh/.pip-accel/sources --build-directory=/tmp/tmpgIpc12 scriptine
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking scriptine
  Running setup.py egg_info for package scriptine

    warning: no previously-included files matching '*.pyc' found under directory 'test'
Successfully downloaded scriptine
2014-11-20 21:34:58 leigh-XPS13 pip_accel[12005] INFO Finished unpacking 1 source distribution in 0.25 seconds.
2014-11-20 21:34:58 leigh-XPS13 pip_accel[12005] INFO Installing from binary distributions ..
2014-11-20 21:34:59 leigh-XPS13 pip_accel.caches.s3[12005] INFO Checking if distribution archive is available in S3 bucket: Linux-x86_643/v7/scriptine:0.2.0:CPython-2.7.tar.gz
2014-11-20 21:34:59 leigh-XPS13 pip_accel.caches.s3[12005] INFO Downloading distribution archive from S3 bucket ..
2014-11-20 21:35:00 leigh-XPS13 pip_accel[12005] INFO Finished installing 1 requirement in 1.37 second.

Forth clean run without PIP_ACCEL_S3_READONLY set and s3 cache warmed

Uninstalling scriptine:
  Successfully uninstalled scriptine
(venv)(s3_readonly)leigh@leigh-XPS13:~/git/pip-accel$ PIP_ACCEL_S3_BUCKET=pip-accel.immun.io PIP_ACCEL_S3_PREFIX="`uname -s`-`uname -m`3"pip-accel install scriptine
install: missing destination file operand after ‘scriptine’
Try 'install --help' for more information.
(venv)(s3_readonly)leigh@leigh-XPS13:~/git/pip-accel$ PIP_ACCEL_S3_BUCKET=pip-accel.immun.io PIP_ACCEL_S3_PREFIX="`uname -s`-`uname -m`3" pip-accel install scriptine
2014-11-20 21:35:18 leigh-XPS13 pip_accel[12045] INFO Unpacking source distribution(s) ..
2014-11-20 21:35:18 leigh-XPS13 pip_accel[12045] INFO Executing command: pip install --no-install --no-index --find-links=file:///home/leigh/.pip-accel/sources --build-directory=/tmp/tmpHQ4qv4 scriptine
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking scriptine
  Running setup.py egg_info for package scriptine

    warning: no previously-included files matching '*.pyc' found under directory 'test'
Successfully downloaded scriptine
2014-11-20 21:35:18 leigh-XPS13 pip_accel[12045] INFO Finished unpacking 1 source distribution in 0.19 seconds.
2014-11-20 21:35:18 leigh-XPS13 pip_accel[12045] INFO Installing from binary distributions ..
2014-11-20 21:35:19 leigh-XPS13 pip_accel.caches.s3[12045] INFO Checking if distribution archive is available in S3 bucket: Linux-x86_643/v7/scriptine:0.2.0:CPython-2.7.tar.gz
2014-11-20 21:35:19 leigh-XPS13 pip_accel.caches.s3[12045] INFO Downloading distribution archive from S3 bucket ..
2014-11-20 21:35:20 leigh-XPS13 pip_accel[12045] INFO Finished installing 1 requirement in 1.08 second.
coveralls commented 9 years ago

Coverage Status

Coverage decreased (-0.41%) when pulling 1eb53f706974cb06a9413028a7e897e38c7dc2c1 on lgtml:s3_readonly into ffe1f19e5a1012b3a92eb2805be087859bdad9b3 on paylogic:master.

lgtml commented 9 years ago

I am sorry coveralls I tried :(

coveralls commented 9 years ago

Coverage Status

Coverage decreased (-0.04%) when pulling 79071390814cdc54e29cc33d3370635945dfff3d on lgtml:s3_readonly into ffe1f19e5a1012b3a92eb2805be087859bdad9b3 on paylogic:master.

coveralls commented 9 years ago

Coverage Status

Coverage decreased (-0.41%) when pulling 3c09ee119274f58ce2a59365c02f56c286965a10 on lgtml:s3_readonly into ffe1f19e5a1012b3a92eb2805be087859bdad9b3 on paylogic:master.

coveralls commented 9 years ago

Coverage Status

Coverage decreased (-0.41%) when pulling 1eda3a549056133a6c7a02cc5f418e303edc3dd1 on lgtml:s3_readonly into ffe1f19e5a1012b3a92eb2805be087859bdad9b3 on paylogic:master.

xolox commented 9 years ago

Hi Leigh and thanks for the pull request! Your use case makes sense so I just merged this. I made some insignificant changes during the merge (e.g. in #41 I added support for configuration files so I gave this new feature a configuration option as well) but the feature is there now and uses the same environment variable.

lgtml commented 9 years ago

Awesome thanks bud!