Install poetry in self-hosted github runner to allow different python versions

[miandreu]

Describe the bug

When using install-poetry in a self-hosted github runner the poetry installation is located in /home/ssm-user/.local/share/pypoetry causing issues if we want to have two branches with different python versions. Here is the issue:

Setting Poetry installation path as /home/ssm-user/.local

Installing Poetry 👷

Retrieving Poetry metadata

The latest version (1.7.1) is already installed.
/home/ssm-user/.local/share/pypoetry/venv/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory
Error: Process completed with exit code 127.

Steps to reproduce the behavior:

Having this configuration:

    #----------------------------------------------
    #  -----  Set-up python  --------------------
    #----------------------------------------------
    - name: Set up python
      id: setup-python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    #----------------------------------------------
    #  -----  Install & configure poetry  -------
    #----------------------------------------------
    - name: Install Poetry
      uses: snok/install-poetry@v1.3.4
      with:
        version: 1.7.1
        virtualenvs-create: true
        virtualenvs-in-project: true
        installer-parallel: true

• Create a branch with python3.10 and run github actions
• Create another one with python3.11 and run github actions
• Try again to run the branch with 3.10 and it will fail with this issue:

/home/ssm-user/.local/share/pypoetry/venv/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

Maybe there is a way to allow different poetry python versions in the same repo but couldn't find how to do it

[sondrelg]

Cool, yeah I can see how that'd be annoying! I think one way to fix this would be to use the $POETRY_HOME env var to make the installation happen in the appropriate runner-dir. Isn't it /runner/_work that's meant for ephemeral installs with self-hosted runners?

I guess you should also be able to cache the install itself, as long as you use the python version in the cache key. Think this section still covers how 👍

[miandreu]

We were using cache before but when using self-hosted it was not working properly and we had problems with the cache

[sondrelg]

What did the cache definition look like?

[miandreu]

This:

    #----------------------------------------------
    #  -----  Install & configure poetry  -------
    #----------------------------------------------
    - name: Install Poetry
      uses: snok/install-poetry@v1
      with:
        version: ${{ env.POETRY_VERSION }}
        virtualenvs-create: true
        virtualenvs-in-project: true
        installer-parallel: true

    #----------------------------------------------
    #       load cached
    #----------------------------------------------
    - name: Set up cache
      id: cached-poetry-dependencies
      uses: actions/cache@v3
      with:
        path: .venv
        key: venv-${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ hashFiles('**/poetry.lock') }}

    #----------------------------------------------
    #  ------  Install Dependencies  ------------
    #----------------------------------------------
    - name: Install dependencies
      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
      run: poetry install

but the issue we are having now is when trying to install poetry, not the dependencies

[sondrelg]

Do you use the setup-python action? If so, I'd propose trying something like this:

      - name: Set up python
        id: setup-python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

+    - name: Load cached Poetry installation
+      id: cached-poetry
+      uses: actions/cache@v3
+      with:
+        path: <your install path>
+        key: poetry-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}

    - name: Install Poetry
      uses: snok/install-poetry@v1
      with:
        version: ${{ env.POETRY_VERSION }}
        virtualenvs-create: true
        virtualenvs-in-project: true
        installer-parallel: true
+   if: steps.cached-poetry.outputs.cache-hit != 'true'

    - name: Set up cache
      id: cached-poetry-dependencies
      uses: actions/cache@v3
      with:
        path: .venv
+      key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

    - name: Install dependencies
      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
      run: poetry install

[sondrelg]

${{ steps.setup-python.outputs.python-version }} contains the patch version as well, which can prevent your venvs from breaking when Github actions releases new minor versions

[miandreu]

Using this I still have the same issue:

The latest version (1.7.1) is already installed.
/home/ssm-user/.local/share/pypoetry/venv/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

[miandreu]

I think I found the solution using this in my ci.yml:

env:
  PYTHON_VERSION: 3.11.7
  POETRY_VERSION: 1.7.1
  POETRY_HOME: /home/ssm-user/.local/share/${PYTHON_VERSION}

so for different branches with different python versions works

[MalteEbner]

I fixed it by installing poetry in a runner-specific directory:

    - name: Install Poetry
      uses: snok/install-poetry@v1
      env:
        # Install poetry in a non-global directory.
        # See https://github.com/snok/install-poetry/issues/141
        POETRY_HOME: ${{ runner.tool_cache }}