pipx: command not found in v3 action

tiagovrtr commented 7 months ago

v3 action fails because pipx is not on PATH

name: Upload Python Package

on:
  pull_request:
    branches:
      - main
    types: [opened, synchronize]
  release:
    types: [created, published]
  workflow_dispatch:

jobs:
  publish:
    runs-on: ubuntu-20.04
    permissions:
      contents: "read"
      id-token: "write"
      pull-requests: "write"
    env:
      POETRY_VERSION: 1.5.0
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version-file: ".python-version"
          cache-dependency-path: "poetry.lock"
      - name: Setup poetry
        uses: abatilo/actions-poetry@v3
        with:
          poetry-version: ${{ env.POETRY_VERSION }}

Run abatilo/actions-poetry@v3
Run pip install --user pipx
Collecting pipx
  Downloading pipx-1.4.3-py3-none-any.whl.metadata (17 kB)
Collecting argcomplete>=1.9.4 (from pipx)
  Downloading argcomplete-3.2.1-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: packaging>=20 in /tmp/_tool/Python/3.10.13/x64/lib/python3.10/site-packages (from pipx) (23.2)
Requirement already satisfied: platformdirs>=2.1 in /tmp/_tool/Python/3.10.13/x64/lib/python3.10/site-packages (from pipx) (3.[11](https://github.com/digestaid/ai-classifier/actions/runs/7598094134/job/20693690876?pr=366#step:4:12).0)
Requirement already satisfied: tomli in /tmp/_tool/Python/3.10.13/x64/lib/python3.10/site-packages (from pipx) (2.0.1)
Collecting userpath!=1.9.0,>=1.6 (from pipx)
  Downloading userpath-1.9.1-py3-none-any.whl.metadata (3.0 kB)
Collecting click (from userpath!=1.9.0,>=1.6->pipx)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Downloading pipx-1.4.3-py3-none-any.whl (65 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.1/65.1 kB 3.6 MB/s eta 0:00:00
Downloading argcomplete-3.2.1-py3-none-any.whl (42 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.3/42.3 kB 4.1 MB/s eta 0:00:00
Downloading userpath-1.9.1-py3-none-any.whl (9.1 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 6.2 MB/s eta 0:00:00
Installing collected packages: click, argcomplete, userpath, pipx
  WARNING: The script userpath is installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pipx is installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed argcomplete-3.2.1 click-8.1.7 pipx-1.4.3 userpath-1.9.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Notice:  A new release of pip is available: [23](https://github.com/digestaid/ai-classifier/actions/runs/7598094134/job/20693690876?pr=366#step:4:25).0.1 -> 23.3.2
Notice:  To update, run: pip install --upgrade pip
/tmp/_temp/9ce4d60b-ea35-4dc7-b344-c0[26](https://github.com/digestaid/ai-classifier/actions/runs/7598094134/job/20693690876?pr=366#step:4:28)0ba88c0f.sh: line 2: pipx: command not found

abatilo commented 7 months ago

@NargiT Any thoughts?

abatilo commented 7 months ago

Hi @tiagovrtr

I tried to reproduce your problem and could not: https://github.com/abatilo/actions-poetry/pull/69/files

I noticed that your logs don't show any output from the pipx ensurepath command. Do you have any ideas why that might be the case?

tiagovrtr commented 7 months ago

Thanks for the quick response. I've realised I might have obfuscated the cause of the issue here. I noticed I had a step using google-github-actions/setup-gcloud. If I remove it the issue disappears. There might be some implications to PATH from gcloud's python


name: Upload Python Package

on:
  pull_request:
    @@ -32,8 +24,6 @@ jobs:
      contents: "read"
      id-token: "write"
      pull-requests: "write"
    env:
      POETRY_VERSION: 1.5.0
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Authenticate to Google Cloud
        id: auth
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{vars.GCP_WORKLOAD_IDENTITY_PROVIDER}}
          service_account: ${{vars.GCP_SERVICE_ACCOUNT}}
          token_format: "access_token"
          access_token_lifetime: "300s"
          create_credentials_file: false
      - name: "Set up Cloud SDK"
        uses: google-github-actions/setup-gcloud@v2
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version-file: ".python-version"
          cache-dependency-path: "poetry.lock"
      - name: Setup poetry
        uses: abatilo/actions-poetry@v3
        with:
          poetry-version: ${{ env.POETRY_VERSION }}

abatilo commented 7 months ago

I believe that would be an implication on the setup-gcloud action, not this one. Or would you disagree? That action is impacting the ability of other actions to run, not the other way around.

tiagovrtr commented 7 months ago

Yes, absolutely, this is probably caused by gcloud's python. Closing the issue

tiagovrtr commented 7 months ago

This might actually have issues unrelated to gcloud, as I get the same error in a clean sef-hosted runner. Also I've noticed the action follows installation steps for macOS, but setting up pipx should depend on the runner OS. In my case it should follow linux steps as the job runs-on: ubuntu-20.04.

python3 -m pip install --user pipx
python3 -m pipx ensurepath

The command that seems to fail is pipx ensurepath, and in the ubuntu installation it is required to prepend it with python3 -m to execute it with the python installation in PATH. Since this action is missing this, the shell might not find pipx as easily as it does in macOS. I will share new developments as I dig into this

NargiT commented 7 months ago

Hello @abatilo,

I just saw your message and @tiagovrtr doc seems correct. depending of the OS the installation steps are not the same.

Should we add os detection and install pipx accordingly ?

abatilo commented 7 months ago

Please do @NargiT. Do you have the bandwidth to work on this?

NargiT commented 6 months ago

sorry not currently. Nevertheless, I would like to know how many OS do you want to support ?

tiagovrtr commented 5 months ago

Please do @NargiT. Do you have the bandwidth to work on this?

Hi @abatilo , I've issued a fix that should cover not only ubuntu but the majority of OS's. In ubuntu 23.xx, pipx can be installed through apt-get install, but that will only be necessary when GitHub launches ubuntu 23.xx runners

atkinsonm commented 5 months ago

Also experiencing this on Ubuntu 18.04. My logs indicate that pip install --user pipx and pipx ensurepath ran.

Run abatilo/actions-poetry@v3
  with:
    poetry-version: 1.5.0
  env:
    pythonLocation: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    PKG_CONFIG_PATH: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib/pkgconfig
    Python_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    Python[2](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:2)_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/[3](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:3).9.16/x6[4](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:4)
    Python3_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    LD_LIBRARY_PATH: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib
Run pip install --user pipx
  pip install --user pipx
  pipx ensurepath
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    PKG_CONFIG_PATH: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib/pkgconfig
    Python_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    Python2_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    Python3_ROOT_DIR: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64
    LD_LIBRARY_PATH: /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib
Collecting pipx
  Downloading pipx-1.4.3-py3-none-any.whl.metadata (17 kB)
Collecting argcomplete>=1.9.4 (from pipx)
  Using cached argcomplete-3.2.3-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: packaging>=20 in /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib/python3.9/site-packages (from pipx) (23.2)
Requirement already satisfied: platformdirs>=2.1 in /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib/python3.9/site-packages (from pipx) (3.11.0)
Requirement already satisfied: tomli in /mnt/github_actions_runner/_work/_tool/Python/3.9.16/x64/lib/python3.9/site-packages (from pipx) (2.0.1)
Collecting userpath!=1.9.0,>=1.6 (from pipx)
  Downloading userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
Collecting click (from userpath!=1.9.0,>=1.6->pipx)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Downloading pipx-1.4.3-py3-none-any.whl (6[5](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:5) kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [6](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:6)5.1/65.1 kB 2.4 MB/s eta 0:00:00
Using cached argcomplete-3.2.3-py3-none-any.whl (42 kB)
Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
Using cached click-8.1.[7](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:7)-py3-none-any.whl (97 kB)
Installing collected packages: click, argcomplete, userpath, pipx
  WARNING: The script userpath is installed in '/mnt/github_actions_runner/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pipx is installed in '/mnt/github_actions_runner/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed argcomplete-3.2.3 click-[8](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:8).1.7 pipx-1.4.3 userpath-1.[9](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:9).2

Notice:  A new release of pip is available: 22.0.4 -> 24.0
Notice:  To update, run: pip install --upgrade pip
/mnt/github_actions_runner/_work/_temp/e3354f2b-245d-4f6c-9[19](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:20)d-da0e86[26](https://slack-github.com/slack/slack-cli-py-tools/actions/runs/336934/job/458155#step:4:28)56ff.sh: line 2: pipx: command not found

Relevant line being WARNING: The script pipx is installed in '/mnt/github_actions_runner/.local/bin' which is not on PATH.

anderssonjohan commented 5 months ago

I was setting up a test repo to try out v3 to see how the switch to pipx would affect us. I was surprised of the installation attempt of pipx.

It only takes a second, but perhaps it's good to add a check to see if pipx is on the path before calling pip install pipx?

On github hosted runners pipx is already in the path:

/home/runner/.local/bin is already in PATH.
/opt/pipx_bin is already in PATH.

⚠️  All pipx binary directories have been added to PATH. If you are sure you
want to proceed, try again with the '--force' flag.

Otherwise pipx is ready to go! ✨ 🌟 ✨

(Output from my test repo here: https://github.com/anderssonjohan/pipx-python-version-github-actions/actions/runs/8434367775/job/23097437766)

pipx v1.4.3 is already included in the runner-images, so I'm not sure why this action has to do it. I've not tested the v3 on older self-hosted linux+windows runners though, and I guess that's different, as mentioned in the comments here.

ra-albrecht commented 1 month ago

I think, the workflow isn't well designed.

Run pip install --user pipx

Collecting pipx
  Downloading pipx-1.6.0-py3-none-any.whl.metadata (18 kB)
Collecting argcomplete>=1.9.4 (from pipx)
  Downloading argcomplete-3.4.0-py3-none-any.whl.metadata (16 kB)
Collecting packaging>=20 (from pipx)
  Downloading packaging-[24](https://github.compute.gurobi.com/IT/vcenter-inventory/actions/runs/5729/job/11596#step:4:26).1-py3-none-any.whl.metadata (3.2 kB)
Collecting platformdirs>=2.1 (from pipx)
  Downloading platformdirs-4.2.2-py3-none-any.whl.metadata (11 kB)
Collecting userpath!=1.9,>=1.6 (from pipx)
  Downloading userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
Collecting click (from userpath!=1.9,>=1.6->pipx)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Downloading pipx-1.6.0-py3-none-any.whl (77 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.8/77.8 kB 8.8 MB/s eta 0:00:00
Downloading argcomplete-3.4.0-py3-none-any.whl (42 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 7.0 MB/s eta 0:00:00
Downloading packaging-24.1-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 kB 7.6 MB/s eta 0:00:00
Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 9.1 MB/s eta 0:00:00
Installing collected packages: platformdirs, packaging, click, argcomplete, userpath, pipx
  WARNING: The script userpath is installed in '/home/docker/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pipx is installed in '/home/docker/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed argcomplete-3.4.0 click-8.1.7 packaging-24.1 pipx-1.6.0 platformdirs-4.2.2 userpath-1.9.2
/home/docker/actions-runner/_work/_temp/289c67f9-3caa-444d-a744-359fcd2bf104.sh: line 2: pipx: command not found
Error: Process completed with exit code 1[27](https://github.XXX/actions/runs/5729/job/11596#step:4:29).

When using a runner on a host where pipx is installed in system it will ALWAYS using the SYSTEM pipx even it is installed localy to user. At the end pipx must be called with full installation path, eg, in case above with /home/docker/.local/bin/pipx for stable work, or $HOME/.local/bin must be enforced to add to PATH.

Currently I have no idea how to solve this in a clean way in docker-based runners (or other runners where pipx isn't pre-installed). With version 2 (pure pip install of poetry) there is no problem overall and I can scale via docker containers as I need.

To be honest: I don't fully understand why pipx must be used for poetry, but I think I just miss something.

ra-albrecht commented 1 month ago

I tried something different, e.g.,. created an environment where $HOME/.local/bin is in the path. This does not work, too :( The reason is simple:

The first run works as expected, e.g., installing pipx - with a specific Python version:

Downloading pipx-1.6.0-py3-none-any.whl (77 kB)
Downloading argcomplete-3.4.0-py3-none-any.whl (42 kB)
Downloading packaging-24.1-py3-none-any.whl (53 kB)
Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
Installing collected packages: platformdirs, packaging, click, argcomplete, userpath, pipx
Successfully installed argcomplete-3.4.0 click-8.1.7 packaging-24.1 pipx-1.6.0 platformdirs-4.2.2 userpath-1.9.2
/home/docker/.local/bin is already in PATH.

poetry config virtualenvs.create true --local
  poetry config virtualenvs.in-project true --local
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/docker/actions-runner/_work/_tool/Python/[3](https://github.compute.gurobi.com/IT/vcenter-inventory/actions/runs/5993/job/12074#step:5:3).11.9/arm64
    PKG_CONFIG_PATH: /home/docker/actions-runner/_work/_tool/Python/3.11.9/arm6[4](https://github.compute.gurobi.com/IT/vcenter-inventory/actions/runs/5993/job/12074#step:5:4)/lib/pkgconfig
    Python_ROOT_DIR: /home/docker/actions-runner/_work/_tool/Python/3.11.9/arm[6](https://github.compute.gurobi.com/IT/vcenter-inventory/actions/runs/5993/job/12074#step:5:6)4
    Python2_ROOT_DIR: /home/docker/actions-runner/_work/_tool/Python/3.11.9/arm64
    Python3_ROOT_DIR: /home/docker/actions-runner/_work/_tool/Python/3.11.9/arm64
    LD_LIBRARY_PATH: /home/docker/actions-runner/_work/_tool/Python/3.11.9/arm64/lib

And tests runs as expected. But: when you run it now with a different Python version, it fails entirely:

/home/docker/.local/share/pipx/venvs/poetry/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory
Error: Process completed with exit code 127.

When removing $HOME/.local/bin from PATH environment it fails again with not finding poetry.

IMHO:

the action should check if a global pipx is already installed and use it when in PATH instead of installing own
when installing local pipx it should be cleaned after the run, eg, removed from $HOME/.local/bin because it will happen that the python version it is installed with is not available on next runs.

Thanks for reading this - and thanks for the nice action

abatilo commented 1 month ago

Thanks for digging into this @ra-albrecht. Are you interested in contributing the changes?

abatilo / actions-poetry

pipx: command not found in v3 action #68