search

CircleCI-Public / python-orb

Common CircleCI tasks for the Python programming language.
https://circleci.com/developer/orbs/orb/circleci/python
MIT License
13 stars 37 forks source link

Cached dependencies not persisting in environment despite successful loading #30

Closed nicolasahar closed 4 years ago

nicolasahar commented 4 years ago

I'm caching my dependencies with python orb (v0.2.1) and I get "command not found" errors for dependencies that are included in my requirements.txt file after I load the cache.

Specifically, my coverage and black commands are not found when they are run. The caching seems to work (see output below), but the installed pip packages do not persist in the environment and cannot be used in downstream run commands, unless I completely remove the caching and manually install my repo with sudo pip install .[dev] (which is too resource-intensive).

I've tried other orb versions, including 0.1.9, 0.2.2, and 0.3.0 but I get the same error.

Am I using the cache improperly? Or is there something else I'm missing?

My config.yml file: 

version: 2.1

orbs:
  python: circleci/python@0.2.1

jobs:
  check_code_quality:
    executor: python/default
    working_directory: ~/shmodels
    steps:
      - checkout
      - python/load-cache
      - python/install-deps
      - python/save-cache
      - run: black --check --line-length 79 --target-version py37 shmodels tests
      - run: ./dev_tools.py lint

  test_all_send_cov:
    executor: python/default
    working_directory: ~/shmodels
    steps:
      - checkout
      - python/load-cache
      - python/install-deps
      - python/save-cache
      - run: coverage run --source=. -m pytest tests/ --verbose
      - run: coverage xml
      - run: bash <(curl -Ls https://coverage.codacy.com/get.sh) report -l python -r coverage.xml 

workflows:
  version: 2
  build_and_test: 
    jobs:
      - check_code_quality
      - test_all_send_cov

Output for run step coverage run --source=. -m pytest tests/ --verbose:

#!/bin/bash -eo pipefail
coverage run --source=. -m pytest tests/ --verbose
/bin/bash: coverage: command not found

Exited with code exit status 127
CircleCI received exit code 127

Output for run step coverage run --source=. -m pytest tests/ --verbose:

#!/bin/bash -eo pipefail
black --check --line-length 79 --target-version py37 shmodels tests
/bin/bash: black: command not found

Exited with code exit status 127
CircleCI received exit code 127

And finally, confirmation that my dependencies (including black and coverage) are indeed installed:

Step python/install-deps:

#!/bin/bash -eo pipefail
if true; then
  pip install --user -r requirements.txt
else
  pip install -r requirements.txt
fi

Output for step python/install-deps:

Requirement already satisfied: allennlp==1.0.0rc5.dev20200521 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (1.0.0rc5.dev20200521)
Requirement already satisfied: appdirs==1.4.4 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (1.4.4)
Requirement already satisfied: appnope==0.1.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (0.1.0)
Requirement already satisfied: attrs==19.3.0 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (19.3.0)
Requirement already satisfied: backcall==0.1.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (0.1.0)
Requirement already satisfied: black==19.10b0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 6)) (19.10b0)
Requirement already satisfied: blis==0.4.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 7)) (0.4.1)
Requirement already satisfied: boto3==1.13.15 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 8)) (1.13.15)
Requirement already satisfied: botocore==1.16.15 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 9)) (1.16.15)
Requirement already satisfied: catalogue==1.0.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 10)) (1.0.0)
Requirement already satisfied: certifi==2020.4.5.1 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 11)) (2020.4.5.1)
Requirement already satisfied: chardet==3.0.4 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 12)) (3.0.4)
Requirement already satisfied: click==7.1.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 13)) (7.1.2)
Requirement already satisfied: conllu==2.3.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 14)) (2.3.2)
Requirement already satisfied: coverage==5.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 15)) (5.1)
Requirement already satisfied: cymem==2.0.3 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 16)) (2.0.3)
Requirement already satisfied: decorator==4.4.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 17)) (4.4.2)
Requirement already satisfied: docutils==0.15.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 18)) (0.15.2)
Requirement already satisfied: filelock==3.0.12 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 19)) (3.0.12)
Requirement already satisfied: flake8==3.8.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 20)) (3.8.1)
Requirement already satisfied: flaky==3.6.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 21)) (3.6.1)
Requirement already satisfied: future==0.18.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 22)) (0.18.2)
Requirement already satisfied: h5py==2.10.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 23)) (2.10.0)
Requirement already satisfied: idna==2.9 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 24)) (2.9)
Requirement already satisfied: importlib-metadata==1.6.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 25)) (1.6.0)
Requirement already satisfied: ipython==7.14.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 26)) (7.14.0)
Requirement already satisfied: ipython-genutils==0.2.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 27)) (0.2.0)
Requirement already satisfied: isort==4.3.21 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 28)) (4.3.21)
Requirement already satisfied: jedi==0.17.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 29)) (0.17.0)
Requirement already satisfied: jmespath==0.10.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 30)) (0.10.0)
Requirement already satisfied: joblib==0.15.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 31)) (0.15.1)
Requirement already satisfied: jsonnet==0.15.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 32)) (0.15.0)
Requirement already satisfied: jsonpickle==1.4.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 33)) (1.4.1)
Requirement already satisfied: mccabe==0.6.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 34)) (0.6.1)
Requirement already satisfied: more-itertools==8.3.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 35)) (8.3.0)
Requirement already satisfied: murmurhash==1.0.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 36)) (1.0.2)
Requirement already satisfied: mypy==0.770 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 37)) (0.770)
Requirement already satisfied: mypy-extensions==0.4.3 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 38)) (0.4.3)
Requirement already satisfied: nltk==3.5 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 39)) (3.5)
Requirement already satisfied: numpy==1.18.4 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 40)) (1.18.4)
Requirement already satisfied: overrides==2.8.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 41)) (2.8.0)
Requirement already satisfied: packaging==20.4 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 42)) (20.4)
Requirement already satisfied: parso==0.7.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 43)) (0.7.0)
Requirement already satisfied: pathspec==0.8.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 44)) (0.8.0)
Requirement already satisfied: pexpect==4.8.0 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 45)) (4.8.0)
Requirement already satisfied: pickleshare==0.7.5 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 46)) (0.7.5)
Requirement already satisfied: plac==1.1.3 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 47)) (1.1.3)
Requirement already satisfied: pluggy==0.13.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 48)) (0.13.1)
Requirement already satisfied: preshed==3.0.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 49)) (3.0.2)
Requirement already satisfied: prompt-toolkit==3.0.5 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 50)) (3.0.5)
Requirement already satisfied: protobuf==3.12.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 51)) (3.12.1)
Requirement already satisfied: ptyprocess==0.6.0 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 52)) (0.6.0)
Requirement already satisfied: py==1.8.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 53)) (1.8.1)
Requirement already satisfied: pycodestyle==2.6.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 54)) (2.6.0)
Requirement already satisfied: pyflakes==2.2.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 55)) (2.2.0)
Requirement already satisfied: Pygments==2.6.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 56)) (2.6.1)
Requirement already satisfied: pyparsing==2.4.7 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 57)) (2.4.7)
Requirement already satisfied: pytest==5.4.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 58)) (5.4.2)
Requirement already satisfied: python-dateutil==2.8.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 59)) (2.8.1)
Requirement already satisfied: regex==2020.5.14 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 60)) (2020.5.14)
Requirement already satisfied: requests==2.23.0 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 61)) (2.23.0)
Requirement already satisfied: responses==0.10.14 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 62)) (0.10.14)
Requirement already satisfied: s3transfer==0.3.3 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 63)) (0.3.3)
Requirement already satisfied: sacremoses==0.0.43 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 64)) (0.0.43)
Requirement already satisfied: scikit-learn==0.23.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 65)) (0.23.1)
Requirement already satisfied: scipy==1.4.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 66)) (1.4.1)
Requirement already satisfied: semantic-version==2.8.5 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 67)) (2.8.5)
Requirement already satisfied: sentencepiece==0.1.91 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 68)) (0.1.91)
Requirement already satisfied: six==1.15.0 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 69)) (1.15.0)
Requirement already satisfied: spacy==2.2.4 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 70)) (2.2.4)
Requirement already satisfied: srsly==1.0.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 71)) (1.0.2)
Requirement already satisfied: tensorboardX==2.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 72)) (2.0)
Requirement already satisfied: thinc==7.4.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 73)) (7.4.0)
Requirement already satisfied: threadpoolctl==2.0.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 74)) (2.0.0)
Requirement already satisfied: tokenizers==0.7.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 75)) (0.7.0)
Requirement already satisfied: toml==0.10.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 76)) (0.10.1)
Requirement already satisfied: torch==1.5.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 77)) (1.5.0)
Requirement already satisfied: tqdm==4.46.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 78)) (4.46.0)
Requirement already satisfied: traitlets==4.3.3 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 79)) (4.3.3)
Requirement already satisfied: transformers==2.9.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 80)) (2.9.1)
Requirement already satisfied: typed-ast==1.4.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 81)) (1.4.1)
Requirement already satisfied: typing-extensions==3.7.4.2 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 82)) (3.7.4.2)
Requirement already satisfied: urllib3==1.25.9 in /usr/local/lib/python3.8/site-packages (from -r requirements.txt (line 83)) (1.25.9)
Requirement already satisfied: wasabi==0.6.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 84)) (0.6.0)
Requirement already satisfied: wcwidth==0.1.9 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 85)) (0.1.9)
Requirement already satisfied: zipp==3.1.0 in /home/circleci/.local/lib/python3.8/site-packages (from -r requirements.txt (line 86)) (3.1.0)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.8/site-packages (from ipython==7.14.0->-r requirements.txt (line 26)) (46.4.0)
CircleCI received exit code 0
mkly commented 4 years ago

Hello @nicolasahar!

There is an issue with the 0.2.1 version Orb that prevents it from caching the /home/circleci/.local/bin directory.

You have two options available

You can not use the built-in load-cache and save-cache commands and instead cache manually:

       - restore_cache:
          key: v1-{{ checksum "requirements.txt" }}
      - python/install-deps
      - save_cache:
          key: v1-{{ checksum "requirements.txt" }}
          paths:
            - /home/circleci/.local/bin
            - /home/circleci/.local/lib

or you can upgrade to the newer 0.3.0 version which has this issue fixed.

mkly commented 4 years ago

cc @felicianotech Note the lack of a /

mkly commented 4 years ago

@nicolasahar I should add you may be able to update this to version 0.2.2 or the Orb as well. See #26