Prepare next release (version 0.9.0).

[fangerer]

This PR prepares the next release (version 0.9). Once this is merged, I'll do a test by creating a release candidate 0.9.0rc1 or something.

Main changes

Refactor GitHub action Release to PyPI

I've refactor our GitHub action that builds the distributions and uploads them to PyPI. Previously, we used the GitHub Release published event. However, that did not work the last time. So, I switched to the push tag event. This means, whenever we push a tag in form of *.*.* (e.g. 0.9.0) we will build and upload the distributions.

Further, I plan to do the whole action on every PR (maybe just post-merge) but only uploading to Test PyPI. This is still disabled because we don't own https://test.pypi.org/project/hpy. I've asked the current owner to transfer ownership but didn't get an answer yet.

I'm now also using a new GitHub action softprops/action-gh-release@v1 that will also create a release on GitHub.

Release Notes

I, ofc, wrote release notes going through all PRs we merged since 0.0.4

[fangerer]

I don't think we should upload to test pypi on every push. It's not really intended to be a heavily used service, just a place to try things out and I wouldn't want to have many hundreds of builds there.

You are probably right. I just saw it in some examples and thought it is maybe a good idea because whenever I do a release, I realize that the build-and-upload-wheel machinery is broken for some reason.

Instead, I suggest that we add the ability to manually trigger a publish to either pypi test (default) or pypi.

Agreed. I suggest that we also do it with tags. We could have a filter and if a tag is in form, let's say, test-*.*.*, we upload to Test PyPI. I'm not sure if we should also create a GitHub draft release in this case. What do you think?

Instead of tags, we could also use branches. E.g all branches that start with test-release/ will upload to Test PyPI.

[hodgestar]

@fangerer I also don't want lots of test-.... tags. I was thinking we could use the GitHub Actions workflow_dispatch option to run test builds from the GitHub Actions UI.

[fangerer]

Btw. the previous owner of https://test.pypi.org/project/hpy deleted the project but I still cannot upload to it:

403 The user 'fangerer' isn't allowed to upload to project 'hpy'. See  
         https://test.pypi.org/help/#project-name for more information.  

Any ideas why this is the case?

[fangerer]

@hodgestar I've done an update on this PR. Please re-review the changelog and here is a summary of the refactored GH actions:

There are now three workflows related to releases:

1. .github/workflows/build_dists.yml is a reusable workflow that builds the source distribution and the wheels. This workflow is meant to be called by other workflows.
2. .github/workflows/create-gh-release.yml triggers with tags in form of *.*.*. It uses build_dists workflow and will create a GitHub release (also uploading the sdist and wheels). If the tag contains letters "rc", the release will be marked as draft.
3. .github/workflows/release-pypi.yml needs to be manually triggered as suggested. It also uses build_dists and will upload to (Test) PyPI. There is an boolean input official (defaults to false) and if this is true, we will upload to official PyPI. Otherwise to Test PyPI.

I've also fixed the wheel testing. In order to do that, we will now ship the static libraries since they are strictly required by the test suite. We may relax this requirement but we could also use those static libs for building HPy extensions. I think they don't really hurt.

[hodgestar]

@fangerer The docs seem to indicate that uses will run the called workflow again, thus building different wheels for GitHub and PyPI releases, which is not what we want I think.

[hodgestar]

I've also fixed the wheel testing. In order to do that, we will now ship the static libraries since they are strictly required by the test suite.

@fangerer I couldn't figure out what if the diff implements this change. Would you mind explaining a bit more?

[fangerer]

The docs seem to indicate that uses will run the called workflow again, thus building different wheels for GitHub and PyPI releases, which is not what we want I think.

Probably yes but I couldn't figure out (so far) how to share file across different workflow runs. Currently, we would manually trigger the relealse-pypi.yml workflow whereas we want to trigger create-gh-release.yml when pushing a tag. Maybe we give up on the push tag trigger and do everything (again) in one workflow.

I couldn't figure out what if the diff implements this change. Would you mind explaining a bit more?

Shipping the static libs is implemented by the change in file setup.py:

    lib_dir = os.path.join(build_ext.build_lib, 'hpy', 'devel')

We now use the same output directory as command build_ext. By default, the build_clib command will only create temporary artifacts in build/temp.*. We don't need to anything else because we already specify the libraries in the setup and so, they will be built as part of command build.

Testing the wheel is then done by action cibuildhweel. The related specification is:

    CIBW_TEST_REQUIRES: pytest pytest-xdist filelock setuptools>=60.2
    CIBW_TEST_COMMAND: cp -R {project}/test ./ && pytest --basetemp=.tmpdir --ignore=test/hpy_devel -n auto ./test/

cibuildwheel already changes the working directory for running the tests to some temporary dir (see test command docs). Therefore, we copy only HPy's test subdir to the working dir and run the tests. I needed to exclude test/hpy_devel/* because those tests try to determine the HPY_ROOT by looking at __file__ which does not work in this case. But I think it is safe to omit those few tests here.

[fangerer]

@hodgestar btw. here is a run where you can see that tests were actually executed with cibuildhweel: https://github.com/hpyproject/hpy/actions/runs/4796648081/jobs/8532661365

[GalaxySnail]

The docs seem to indicate that uses will run the called workflow again, thus building different wheels for GitHub and PyPI releases, which is not what we want I think.

What about reproducible builds? I'm not sure if it is too difficult or takes a lot of effort to do. But it will be great if we can achieve reproducible builds.

[hodgestar]

@GalaxySnail Reproducible builds would be cool, but let's not try do that in this PR or for 0.9.0.

[fangerer]

@hodgestar Another update: The release-pypi workflow again contains everything (build dists, publish to (Test) PyPI, and create GH release). The workflow needs to be triggered manually. One needs to specify the Git tag for that. Hence, the release tag must be pushed before running the workflow. Are you fine with this?

[hodgestar]

@fangerer Looks great. I made two suggestions to the explanation of the tag and official workflow options.

[fangerer]

It would be nice to be able to build from a branch for testing purposes, but +1 this for now so that it can be merged. We can improve things later as needed.

Agreed. Let's get the release out and I promise to add this ability. We can discuss that in the dev call.