ckanext-xloader: PyPi integration with github actions for tag auto deployment

[duttonw]

Looking at https://pypi.org/project/ckanext-xloader/#history Last release was Released: Dec 1, 2022

This has come up in issues prior e.g. https://github.com/ckan/ckanext-xloader/issues/211

For the current maintainers who have access to this pypi package. Can we have a github actions config setup Or provide user @duttonw (pypi user duttonwz ) and @ThrawnCA access to maintain this package.

https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

cc: @wardi @amercader @davidread

[wardi]

@duttonw I've sent you a pypi invite. @ThrawnCA do you have a pypi account?

[ThrawnCA]

@ThrawnCA do you have a pypi account?

I don't, no.

[brooks-eco]

Here in Sept 2024 installing to 2.11 and its kinda nasty that install instructions advise pip install ckanext-xloader which downloads ckanext-xloader-0.12.2.tar.gz from 2022

[duttonw]

I have been given access to pypi for this module, I'll try put out a release in the coming weeks, looking at cicd GitHub action for doing it automatically.

[duttonw]

https://pypi.org/project/ckanext-xloader/1.1.0/ released manually, will work towards a github action system in due time.

python pre-req for deployment

1. pip install build
2. pip install twine
3. create if not existing .pypirc file

   
   [distutils]
   index-servers =
   pypi
   testpypi

[pypi] repository = https://upload.pypi.org/legacy/ [ckanext-xloader] username = token password = $token

4. Ensure gpg signing on commits is setup.
5. checkout 

git clone https://github.com/ckan/ckanext-xloader.git cd ckanext-xloader/

## Version increment/bump
1. Update setup.py (version to what we bump to)
2. Update Changelog (if possible)
3. Commit and push tag
```git commit -m"chore: update changelog/bump version $version"
git tag -s $version -m "Update version $version"
git push  --follow-tags

Deploy

python -m build
twine upload dist/*
rm -rf dist

[duttonw]

Hi @wardi ,

For this to be completed, i've created the publish system under PR https://github.com/ckan/ckanext-xloader/pull/228

I also need the pypi integration setup to be happy with github auth.

For that, I'm also needing more permissions inside pypi (or yourself or another to configure it).

Details below:

project_name: ckanext-xloader owner: ckan repository: ckanext-xloader workflow_filename: publish.yml environment: pypi provider: github

https://pypi.org/manage/account/publishing/ is for new packages. It needs to be setup at https://pypi.org/manage/project/ckanext-xloader/settings/publishing/

prefilled link for you: here

[duttonw]

Hi @wardi ,

Also need github environments created (Don't have permissions for settings on this)

https://github.com/ckan/ckanext-xloader/settings/environments

Add these Environment name's:

• testpypi
• pypi

We can also restrict who can approve the release to lock down by adding a team or user to "Required reviewers" if required.

You can do this also with gh cli gh api.

basic way is: gh login gh api repos/ckan/ckanext-xloader/environments/pypi --method PUT gh api repos/ckan/ckanext-xloader/environments/testpypi --method PUT

---

On a different note, if we wish to allow github actions to allow a manual workflow on version incrementing (version cutting etc). It's best to have a working account create that has the pgp/ssh private keys loaded into the repo environment (or org).

I've done this with qld-gov-au-bot-ServiceAccount as well as with this script. https://github.com/qld-gov-au/qgds-vanilla/blob/main/.github/workflows/update.yml#L37

At this time, the system i've built for xloader is just pypi publish, the version bump will need to be done with github (release publish) or via a human.

[duttonw]

Yay, its a bit rough around the edges but we have a manual github action workflow

That can publish to pypi without an individuals token in the loop https://github.com/ckan/ckanext-xloader/actions/runs/11358261005/job/31592591256

Run pypa/gh-action-pypi-publish@release/v1
/usr/bin/docker run --name fa19904a785c0c6a9c42a685baa2253e47ea59_b73dfd --label fa1990 --workdir /github/workspace --rm -e "version" -e "INPUT_USER" -e "INPUT_PASSWORD" -e "INPUT_REPOSITORY-URL" -e "INPUT_REPOSITORY_URL" -e "INPUT_PACKAGES-DIR" -e "INPUT_PACKAGES_DIR" -e "INPUT_VERIFY-METADATA" -e "INPUT_VERIFY_METADATA" -e "INPUT_SKIP-EXISTING" -e "INPUT_SKIP_EXISTING" -e "INPUT_VERBOSE" -e "INPUT_PRINT-HASH" -e "INPUT_PRINT_HASH" -e "INPUT_ATTESTATIONS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/ckanext-xloader/ckanext-xloader":"/github/workspace" fa1990:4a785c0c6a9c42a685baa2253e47ea59  "__token__" "" "" "" "" "" "false" "" "false"
Checking dist/ckanext_xloader-1.1.1-py3-none-any.whl: PASSED
Checking dist/ckanext_xloader-1.1.1.tar.gz: PASSED
Uploading distributions to https://upload.pypi.org/legacy/
Uploading ckanext_xloader-1.1.1-py3-none-any.whl
Uploading ckanext_xloader-1.1.1.tar.gz

View at:
https://pypi.org/project/ckanext-xloader/1.1.1/

https://pypi.org/project/ckanext-xloader/1.1.1/

---

Next job is working on workflow bumping that also deploys (so all in one).

I'm eyeing this plugin to provide us the ability to just state if its a major/minor/patch/etc.

https://github.com/callowayproject/bump-my-version?tab=readme-ov-file#github-actions

@wardi For this to work with github action's security in not allowing a runner to kick off other jobs. I can either, inject my own credentials { not advised }. Or use a service account user.

My preference is a service account for this which can be reused for other projects (cookie cutter).

What is your preferred username for it?

I'm thinking ckan-bot-ServiceAccount