Closed sdb9696 closed 1 month ago
Hello @sdb9696 Thank you for creating this issue. We will investigate it and get back to you as soon as we have some feedback.
Hello @sdb9696 – my team ran into this same issue recently. We solved it by pinning the python version in our pyproject.toml
file, under [tool.poetry.dependencies]
. See example below.
[tool.poetry.dependencies]
python = "3.11.4"
Thanks @andwaredev but we run our CI against all supported python versions to check there are no issues so not sure this workaround will help.
I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before setup-python
. It can not be assumed that python is present before setup-python
has run because some runner images will not have python pre-installed.
This is what I would like to see working:
- uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'poetry'
- run: pipx install poetry
It's always struck me as a bad chicken/egg problem that poetry is the only cache type that requires the environment manager to be installed before it can be used. Even pipenv is not required to be installed https://github.com/actions/setup-python/blob/82c7e631bb3cdc910f68e0081d67478d79c6982d/src/cache-distributions/pipenv-cache.ts#L20 it just uses the default pipenv location.
The Poetry default cache location is well known and I bet in CI the vase majority of projects are using that case. The big difference appears to be that the Poetry option is trying to support the fact that Poetry can install virtual environments in the project. Pipenv can also do this but it's just ignored for that option. I'd recommend just caching the default location like Pipenv, or you could have an option for enabling caching in project virtual environments it people use it.
@silverwind You can tell poetry to use a specific version of python prior to running install poetry env use <python executable>
then run poetry install
https://python-poetry.org/docs/managing-environments#switching-between-environments
@evilhamsterman doesn't really help me if there is simply no python
installed before setup-python
ran. Of course one could do workarounds by running setup-python
twice, but I rather skip on enabling cache than doing such hacks.
I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before
setup-python
. It can not be assumed that python is present beforesetup-python
has run because some runner images will not have python pre-installed.This is what I would like to see working:
- uses: actions/setup-python@v5 with: python-version: '3.12' cache: 'poetry' - run: pipx install poetry
This also makes the assumtion that you have pipx which I do not :)
I'd highly recommend to install poetry as part of this python version fetching if cache is set to poetry.
Chances are everyone is going to need poetry if they are using poetry cache, so maybe get the specific python version with poetry pre-installed. This seems to be the only viable solution.
This also makes the assumtion that you have pipx which I do not :)
I think setup-python
does install pip and its variants currently, right?
This also makes the assumtion that you have pipx which I do not :)
I think
setup-python
does install pip and its variants currently, right?
Apparently, it doesn't so to install poetry without pip and having a poetry cache, this would be the ideal:
- uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'poetry'
- uses: snok/install-poetry@v1
This is indeed inconvenient. We install Python and Poetry in the following way:
- name: Install Poetry
run: |
python -m pip install poetry
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.12
cache: poetry
cache-dependency-path: projects/investment_decision_support_system/poetry.lock
- name: Set Poetry environment
run: |
python -m pip install poetry
cd projects/investment_decision_support_system/
poetry env use 3.12
This triggered the following error:
- Installing xmltodict (0.13.0 9c3ec3c)
ChefBuildError
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Traceback (most recent call last):
File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
main()
File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 128, in get_requires_for_build_wheel
backend = _build_backend()
^^^^^^^^^^^^^^^^
File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 70, in _build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
import setuptools.version
File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module>
register_finder(pkgutil.ImpImporter, find_on_path)
^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
at /opt/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
160│
161│ error = ChefBuildError("\n\n".join(message_parts))
162│
163│ if error is not None:
→ 164│ raise error from None
165│
166│ return path
167│
168│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Do note how the installation is failing because Poetry is using Python 3.10 while the specified Python version is 3.12.
This took a while to chase down. I'm not sure what exactly depends on Poetry but perhaps it would be possible to integrate those functions into setup-python instead of having a hard dependency on the whole package?
Alternatively, if there's a workaround it would be great if the documentation could be updated.
Ultimately we're considering just using actions/cache directly to cache poetry rather than setup-python's built in support
I just ran into this issue as well. I used an example in the docs that triggers this bug beautifully. Maybe time to ramp up the tests for this action?
I am running the following workflow which works:
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pipx install poetry==1.8.3
- run: echo "/root/.local/bin" >> $GITHUB_PATH
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: poetry
- run: pipx install poetry==1.8.3
- run: poetry install
working-directory: app
- run: poetry run flake8
working-directory: app
pipx
is smart enough not to install poetry again, and somehow it ends up in the correct environment without any hacks.
What I had to do was add /root/.local/bin
to the $GITHUB_PATH
, because otherwise it wouldn't be able to find poetry
anywhere and fail at actions/setup-python
:
[pr/lint] 💬 ::debug::Found 1 files to hash.
[pr/lint] 💬 ::debug::followSymbolicLinks 'true'
[pr/lint] 💬 ::debug::implicitDescendants 'true'
[pr/lint] 💬 ::debug::matchDirectories 'true'
[pr/lint] 💬 ::debug::omitBrokenSymbolicLinks 'true'
[pr/lint] 💬 ::debug::Search path '/Users/till/Documents/workspaces/project/pdf-test'
[pr/lint] 💬 ::debug::Processing Poetry project at /Users/till/Documents/workspaces/project/pdf-test/app
[pr/lint] ❗ ::error::Unable to locate executable file: poetry. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
[pr/lint] ❌ Failure - Main actions/setup-python@v5
[pr/lint] ⚙ ::set-env:: Python_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint] ⚙ ::set-env:: Python2_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint] ⚙ ::set-env:: Python3_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint] ⚙ ::set-env:: LD_LIBRARY_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib
[pr/lint] ⚙ ::set-env:: pythonLocation=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint] ⚙ ::set-env:: PKG_CONFIG_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
[pr/lint] ⚙ ::set-output:: python-version=3.11.8
[pr/lint] ⚙ ::set-output:: python-path=/opt/hostedtoolcache/Python/3.11.8/x64/bin/python
[pr/lint] ⚙ ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint] ⚙ ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64/bin
[pr/lint] exitcode '1': failure
Hi @sdb9696 , It is the expected behaviour, if the setup python tries to install the python version which was not specified in pyproject.toml file, under [tool.poetry.dependencies], the poetry install use the python version used during poetry installation from runner. Here's the screenshots for your reference.
[tool.poetry.dependencies]
python = "^3.10.9"
numpy = "^1.22.3"
tqdm = {version = "4.66.4", optional = true }
pandas = "^2.2.2"
Hi @sdb9696, Just a gentle reminder regarding this issue, We have updated the document. If you have any questions or need further assistance, Please let us know.
Hi @sdb9696, Just a gentle reminder regarding this issue, We have updated the document. If you have any questions or need further assistance, Please let us know.
Hi @gowridurgad, apologies for the delayed reply I have been away.
I'm not completely sure I follow the expected behaviour point or the point about matching the python version in pyproject.toml.
Your example above is doing poetry env use "${{ steps.setup-python.outputs.python-path }}"
. Is that something you added specifically to your test or has it been added to the setup-python codebase since I opened the original issue? My suggestion was to update the docs to include that suggestion and I can see from above that it will work if that's done.
w.r.t the setup-python interpreter not matching the toml spec that is not the issue I was reporting. Our toml specifies python = ">=3.8.0"
so this isn't the issue I was reporting. The issue is that poetry will not use the setup-python python version unless the user runs poetry env use "${{ steps.setup-python.outputs.python-path }}"
before running install.
Hi @sdb9696, we haven't use the command poetry env use "${{ steps.setup-python.outputs.python-path }}" in workflow file. Poetry seems to be automatically using the Python version set by setup-python, regardless of whether the command is used. However, The Poetry fails to use the setup-python Python version only when there's a mismatch between this version and the one specified in the pyproject.toml file. In such cases, using the poetry env use "${{steps.setup-python.outputs.python-path }}" actually causes the build to fail. here's the screenshots for your reference. screenshot1: with poetry env use "${{ steps.setup-python.outputs.python-path }}" .
screenshot2: without poetry env use "${{ steps.setup-python.outputs.python-path }}"
screenshot3: poetry env use "${{ steps.setup-python.outputs.python-path }}" causes the build to fail if python version didn't match version specified in pyproject.toml file.
I ran into the same issue. The existing workflow stopped working recently, without any changes to it.
I fixed it by changing how poetry is installed.
steps:
- uses: actions/checkout@v4
- name: Install poetry
run: curl -sSL https://install.python-poetry.org | python3 -
- uses: actions/setup-python@v5
with:
python-version: "3.12.3"
cache: "poetry"
- name: Install dependencies
run: poetry install
Hi @andreipetre, Could you please attach link to the build or public repository to reproduce the issue ? If you are referring to the workflow mentioned in the document, It is working correctly. The workflow outlined in the document and the one you mentioned in your previous comment are functioning identically.
@gowridurgad maybe you're missing the issue because you're fixing the python version in your pyproject.toml. Have you tried this with a loose python specifier of >=3.8
? Then poetry will use whichever version it was pipx
installed with and not the version installed by setup_python
Hi @sdb9696, We tried using a loose Python specifier of >=3.8 and Poetry is using the version installed by setup_python. Here is a screenshot for reference.
can you please share your updated workflow link to check and assist further if anything missed as we are not seeing the issue from our end.
[tool.poetry.dependencies]
python = ">=3.8"
requests = "^2.25.1
Hi @gowridurgad, I don't have a workflow link to point you to as I ended up using actions/cache directly in the end. Can you point me to the PR/branch you've used to generate the output above and I'll see what the difference is?
Hi @sdb9696, Here is the link to the repository https://github.com/setup-actions-demo/test-setup-python/actions/runs/9868764478/job/27251296773 Please take a look and let us know if you find any differences or have any questions.
Hi @sdb9696, Just a gentle reminder regarding this issue, If you have any updates, Please let us know.
Hi @sdb9696, Just a gentle reminder regarding this issue, If you have any updates, Please let us know.
Hi, apologies for the delay. I tried to recreate the issue and couldn’t anymore. It’s possible something has fixed it so I’d say good to close with the doc updates. Many thanks for your time with this.
Hello @sdb9696 , The PR has been merged and the documentation has been updated. For reference, you may visit https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages. I am proceeding with closing the issue. Please feel free to contact us in case of any further concerns. Thank You !
Description:
The recommendations for caching poetry dependencies causes the wrong python version to be used for
poetry install
.Because pipx install poetry happens before
setup-python
and By default, Poetry will try to use the Python version used during Poetry’s installation to create the virtual environment for the current project.Maybe the solution here is to update the docs to make sure that users know to run
poetry env use ${{ steps.setup-python.outputs.python-path }}python
prior to runningpoetry install
. I have not tested that works however.Action version:
v5
Platform:
Runner type:
Tools version:
All versions of python affected except the default version installed on the github runner.
Repro steps:
Follow the steps outlined in advance-usage.md for caching poetry dependencies.
Run
poetry install
Expected behavior:
poetry install
should use the version of python set bysetup-python
Actual behavior:
poetry install
uses the default version of python from the runner os.