search

CircleCI-Public / python-orb

Common CircleCI tasks for the Python programming language.
https://circleci.com/developer/orbs/orb/circleci/python
MIT License
13 stars 37 forks source link

Cache doesn't include ~/.local/bin #27

Closed jaqx0r closed 4 years ago

jaqx0r commented 4 years ago

The cache doesn't include the binaries installed by pip, which makes it very confusing when you have a set of commands like:

      - python/load-cache
      - python/install-deps
      # CI-only deps, not in requirements.txt
      - run: pip install --user coveralls yapf pylint
      - run: export PATH=$PATH:$(python -m site --user-base)/bin
      - python/save-cache

and the commands don't exist.

(See https://github.com/CircleCI-Public/python-orb/pull/26)

One fix is to merely add ~/.local/bin to the paths as the above pull does.

A better fix is to get the full installed file list from pip, cp or tar them up into the cache path; on load-cache untar or cp across to the ~/.local path.

Here's an almost complete way to find them:

pip list --user --format json | jq -r '.[].name' | xargs pip show -f

The resulting output has a list of files not easily parseable, and a Location field that needs to be prepended. Binaries also have relative parents pointing back to the bin dir.

jaq@gunstar ~% pip show coveralls -f
Name: coveralls
Version: 1.10.0
Summary: Show coverage stats online via coveralls.io
Home-page: http://github.com/coveralls-clients/coveralls-python
Author: coveralls-clients contributors
Author-email: None
License: MIT
Location: /home/jaq/.local/lib/python2.7/site-packages
Requires: docopt, urllib3, coverage, requests
Required-by: 
Files:
  ../../../bin/coveralls
  coveralls-1.10.0.dist-info/INSTALLER
  coveralls-1.10.0.dist-info/LICENSE.txt
  coveralls-1.10.0.dist-info/METADATA
  coveralls-1.10.0.dist-info/RECORD
  coveralls-1.10.0.dist-info/WHEEL
  coveralls-1.10.0.dist-info/entry_points.txt
  coveralls-1.10.0.dist-info/top_level.txt
  coveralls/__init__.py
  coveralls/__init__.pyc
  coveralls/__main__.py
  coveralls/__main__.pyc
  coveralls/api.py
  coveralls/api.pyc
  coveralls/cli.py
  coveralls/cli.pyc
  coveralls/exception.py
  coveralls/exception.pyc
  coveralls/git.py
  coveralls/git.pyc
  coveralls/reporter.py
  coveralls/reporter.pyc
  coveralls/version.py
  coveralls/version.pyc
FelicianoTech commented 4 years ago

Closed in: https://github.com/CircleCI-Public/python-orb/pull/26

albertyw commented 4 years ago

The v1.0.0 release removed caching of ~/.local/bin causing this issue to reappear. https://github.com/CircleCI-Public/python-orb/commit/3c5e34de69a3276d9a4af738133d17398994ec1b#diff-1ee55a5db0a8722d24725e43c731daa7L19 was removed when save-cache.yaml was rewritten to install-packages.yaml where it should appear at https://github.com/CircleCI-Public/python-orb/blob/master/src/commands/install-packages.yml#L184-L188

dsayling commented 4 years ago

@albertyw just saw your comment. This was similar to another request received via another forum. I opened and explained that with the new convenience images, this should not be an issue with the latest orb: https://github.com/CircleCI-Public/python-orb/issues/52

Please let us know on that issue if there's something wrong with that implementation or something that can be improved.