Using self-hosted runners for publishing

[mike-pisman]

Hi, my publish action get stuck indefenetly waiting for the token:

Notice: Attempting to perform trusted publishing exchange to retrieve a temporary short-lived API token for authentication against https://upload.pypi.org/legacy/ due to __token__ username with no supplied password field

It does not print any additional messages or errors. The job run: https://github.com/unipoll/API/actions/runs/6228064737/job/16906315888

In pyproject.toml I have specified project name as:

[project]
name = "unipoll-api"

Here is the workflow file

pypi:
    name: Publish to PyPI
    needs: release
    runs-on: arc-runner-k8s
    strategy:
      matrix:
        python-version: ['3.11']
    environment:
      name: pypi
      url: https://pypi.org/p/unipoll-api
    permissions:
      id-token: write
    steps:
    - name: Set up Python ${{matrix.python-version}}
      uses: actions/setup-python@v4
      with:
        python-version: ${{matrix.python-version}}
    - name: Install dependencies
      run: |
        python -m pip install build
    - name: Download source code
      uses: actions/checkout@v3
    - name: Build package distributions
      run: |
        python -m build
    - name: Publish package distributions to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      if: needs.release.outputs.released == 'true'

Did I miss some configuration or improperly configured the project?

[mike-pisman]

The issue was the runner, I managed to publish the package by using github runner runs-on: ubuntu-latest. I guess, I cannot use my self-hosted runner for Trusted Publishing. Can I use API token instead?

[webknjaz]

I guess, I cannot use my self-hosted runner for Trusted Publishing.

@woodruffw could you look into this? Quick googling suggests that GitHub doesn't limit OIDC to GH-hosted runners but there's a metadata field that can be used to check if it's self-hosted...

Can I use API token instead?

Sure, that's always worked. While it's de-emphasized in the README now, it's still supported: https://github.com/pypa/gh-action-pypi-publish#specifying-a-different-username. Just put it into a repo secret that I still recommend making under a GitHub Environment rather than a repo-global one. Plus, you can make use of the required approvals this way.

[woodruffw]

I guess, I cannot use my self-hosted runner for Trusted Publishing.

@woodruffw could you look into this? Quick googling suggests that GitHub doesn't limit OIDC to GH-hosted runners but there's a metadata field that can be used to check if it's self-hosted...

Yeah, the relevant field is runner_environment. That being said I don't believe PyPI checks that claim at all, so we should support OIDC identities from self-hosted runners currently (although I have no idea how GitHub produces them internally -- I guess self-hosted runners call back into their IdP infrastructure? That doesn't seem safe to me 🙂)

my publish action get stuck indefenetly waiting for the token:

Unfortunately, this strongly suggests a network hiccup or outage on GitHub's side (or, less likely, PyPI's). I've seen occasional timeouts on OIDC credential retrieval before, and it's possible that self-hosted runners may be more susceptible to them (or that the runner agent doesn't have a timeout in place).

(If the error was due to an invalid token from the self-hosted runner, you would have instead seen a detailed error message from PyPI explaining which claims were invalid.)

[webknjaz]

Alright, looking like there's nothing actionable on our side.

[mike-pisman]

I tried to use API token, but the upload gets stuck at 0%. Seems, that @woodruffw is right, and there is some connection issues, as the container struggles to send the data to PyPI server. I checked the log but didn't find anything useful. Job: https://github.com/unipoll/API/actions/runs/6240377666/job/16940353534

I have attached the log from the container, in case anyone is interested. Guess, I will have to reserve to using GitHub runners for uploading.

@webknjaz, @woodruffw thank you for looking into the issue anyway. log.txt

[woodruffw]

Does your self-hosted runner happen to be behind a corporate firewall or something similar? Many companies block PyPI; that might be a proximate cause here.

[mike-pisman]

I host my runners in the home lab behind OPNsense firewall. I also though that maybe it's the firewall blocking the connection, but after checking the firewall log(while the action was running) I didn't see anything abnormal. There shouldn't be anything blocking the connection. Besides, I am successfully able to run actions to make GitHub releases or build and upload images to Docker Hub. So, I'm not sure whether it's the firewall issue or maybe some services like HAproxy get in the way, or maybe even my internet provider... but I think that would be too difficult for me to investigate, since I'm only learning all these devops stuff. :smiley:

So we'll have to wait for someone else(more knowledgeable) to confirm whether the are issues running pypi-publish action on self-hosted GitHub Action Runner Controller or it's just my misconfiguration causes the problem.