search

CircleCI-Public / python-orb

Common CircleCI tasks for the Python programming language.
https://circleci.com/developer/orbs/orb/circleci/python
MIT License
13 stars 37 forks source link

Include /home/circleci/.local/bin/ in the cache #52

Closed dsayling closed 3 years ago

dsayling commented 3 years ago

Describe Request:

From the ideas board, Ben asked:

When installing tools like isort, black or bandit the executable is dropped into that folder. The cache will think it's installed but since the bin folder isn't restored the command will be missing.

dsayling commented 3 years ago

In the latest orb and convenience image, there seems to be no problem relinking the executable. Below is an example where black is installed, the executable removed, installed again (from cache), executable available again.

circleci@a6664f24835e:~/project$ black
No Path provided. Nothing to do 😴
circleci@a6664f24835e:~/project$ rm $(readlink -f $(which black))
circleci@a6664f24835e:~/project$ black
bash: /home/circleci/.pyenv/shims/black: No such file or directory
circleci@a6664f24835e:~/project$ pip install black
Requirement already satisfied: black in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (20.8b1)
Requirement already satisfied: typed-ast>=1.4.0 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (1.4.1)
Requirement already satisfied: pathspec<1,>=0.6 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (0.8.0)
Requirement already satisfied: appdirs in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (1.4.4)
Requirement already satisfied: regex>=2020.1.8 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (2020.9.27)
Requirement already satisfied: typing-extensions>=3.7.4 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (3.7.4.3)
Requirement already satisfied: click>=7.1.2 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (7.1.2)
Requirement already satisfied: toml>=0.10.1 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (0.10.1)
Requirement already satisfied: mypy-extensions>=0.4.3 in /home/circleci/.pyenv/versions/3.8.6/lib/python3.8/site-packages (from black) (0.4.3)
WARNING: You are using pip version 20.2.1; however, version 20.2.3 is available.
You should consider upgrading via the '/home/circleci/.pyenv/versions/3.8.6/bin/python3.8 -m pip install --upgrade pip' command.
circleci@a6664f24835e:~/project$ black
No Path provided. Nothing to do 😴

The python orb tests this with pytest in most of the jobs. The cache is restored, pip install runs again, but runs from cache, the executable pytest is available.

albertyw commented 3 years ago

This example works with installing dependencies into .pyenv. However, in my case it looks like .pyenv is ignored and all packages and bin symlinks are stored in ~/.local. I'm using orb circleci/python@1.2.0 and my install-packages config looks like:

      - python/install-packages:
          pkg-manager: pip 
          pip-dependency-file: requirements-test.txt
dsayling commented 3 years ago

@albertyw can you provide the contents of your req.txt or your full config so I can try to understand/reproduce the error. The job here covers this scenario with pytest, but I suppose some other executable could be broken.

Also, what image are you using? This orb is inteded to work with the next gen images where pyenv is used: https://circleci.com/developer/images/image/cimg/python