search

apache / datafusion-python

Apache DataFusion Python Bindings
https://datafusion.apache.org/python
Apache License 2.0
374 stars 78 forks source link

Add GitHub action to publish wheels to PyPi / TestPyPi #336

Open andygrove opened 1 year ago

andygrove commented 1 year ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. We already have a GitHub workflow that builds wheels when the repo is tagged with a release tag or rc tag. Hoever, we have to manually download these files and upload them to PyPi and TestPyPi.

Describe the solution you'd like I would like this automated.

Describe alternatives you've considered Keep doing it manually

Additional context

kylebrooks-8451 commented 1 year ago

Do we only want an upload to PyPi to occur if there is a manual approval? If so, we could make a Github environment that requires an approval and add it to the commented out workflow code that uploads to PyPi to ensure releases still have a vote. Maybe pypi is a good name for the env. Are you looking for help on this? If so, we can help out.

andygrove commented 1 year ago

@kylebrooks-8451 yes, we want to publish when certain tags are pushed to the repo (similar to how we currently build the wheels when rc tags are pushed.

For example, pushing 23.0.0-rc1 should trigger an upload to TestPyPi. Since we already run maturin build on rc tags, maybe we just need to update this to `maturin publish instead? We will need to store an authentication token as a GitHub secret.

For the final release, I think we can just have a script that downloads the wheels from TestPyPi and uploads them to PyPi. No need to build new artifacts. What do you think?

andygrove commented 1 year ago

Are you looking for help on this? If so, we can help out.

Yes, that would be fantastic. Thank you.

andygrove commented 1 year ago

One challenge is how we handle the case where we have multiple release candidates, such as 23.0.0-rc1 followed by 23.0.0-rc2. I don't know if we can re-upload wheels and overwrite them since the wheel version would still just be 23.0.0?

ayushdg commented 1 year ago

pypi uses the rcN part of the tag to identify that a package is a release candidate and usually marks it as such. So if the wheel version has the rc tag, it doesn't overwrite them but instead creates a new entry. There's more info in pep440 and here's an example of how it shows up on pypi for a project. image