search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.51k stars 3.02k forks source link

Automate the Release Process #2314

Closed dstufft closed 3 years ago

dstufft commented 9 years ago

This issue depends on https://github.com/pypa/pip/issues/2310, https://github.com/pypa/pip/issues/2312, and https://github.com/pypa/pip/issues/2313.

It would be really great to automate the entire release process. Currently there are a bunch of manual steps and the issues https://github.com/pypa/pip/issues/2310, https://github.com/pypa/pip/issues/2312, and https://github.com/pypa/pip/issues/2313 capture the details of automating those steps. However that still requires that someone with the keys to release installs the dependencies for doing a release and invokes the do-the-release command. It's possible that we consider that good enough and this issue should just be closed as won't fix. However the openstack folks have what I think is a fairly nice solution, it essentially boils down to doing a release is as simple as git tag -s X.Y.Z && git push --tags. From there on our their automation does everything else.

The biggest downside to this (besides the fact we have to run a server that does the automation) is that the person doing the release only signs the git tag. They will not be signing the artifacts that get uploaded to PyPI. We can have a key that lives on the automation server that signs the artifacts before we upload them. The biggest difference between that is one could make an argument that doesn't hold the same level of guarantee as the release artifacts being signed by a person does. To be specific, this is about the gpg signatures that get uploaded to PyPI which is primarily used by people like Linux distributions to verifying the downloads from PyPI.

pradyunsg commented 7 years ago

Thoughts on an interactive script that does this for you?

Basically,

AraHaan commented 7 years ago

You can also do like now aiohttp does and use Appveyor and Travis CI for the pypi stuff. After all it can encrypt the pypi user name and password. Same for AppVeyor so it should be no troubles doing it. @asvetlov can also help you set it up as well for automatic deployment on PyPi and it also does the tag stuff for you if you set it up right. Also reading the scripts they use with their aiohttp can help as well for the CI's.

pradyunsg commented 7 years ago

@AraHaan Thanks for the pointer! I'll take a look. :)

pradyunsg commented 7 years ago

Generate artifacts + Upload

Just to be extra clear, this step takes place on the CI servers, not locally, and is indeed triggered by the tagging of the release. At least, that's how I thought of it.

asvetlov commented 7 years ago

Feel free to ask me if you have any problem. I spent some time tuning autodeployment for aiohttp and other aio libraries but now it works perfectly.

pradyunsg commented 7 years ago

@asvetlov Will do. ^.^

pradyunsg commented 7 years ago

We could adopt a process like numpy? [1]

Build the wheels on CI, upload them to some online storage, download, sign and upload.

This process could be made into a script that performs these steps, needing only input from the releaser only for signing the release - the signing key would be a parameter and gpg would ask for passphrase. That keeps the same level of guarantee for the release artifacts and the automation would handle everything except the signing...

PS: Sorry for the cross-post.

RonnyPfannschmidt commented 7 years ago

@dstufft i would like to propose a combination of repeatable builds and allowing other people to upload signatures, then anyone who cares can upload a signature of artifacts they created themselfes

i'd love to see pip releases signed by 5-10 people arriving at the artifacts independently and not just a single person

dstufft commented 7 years ago

I'm more likely to just stop uploading GPG signatures than I am to invest any effort in adding additional features to supporting it. Long term TUF is the future here and it does support N of M signatures.

asvetlov commented 7 years ago

Sorry, what is TUF?

pradyunsg commented 7 years ago

@asvetlov The Update Framework - A Framework for Securing Software Update Systems

https://theupdateframework.github.io

pradyunsg commented 4 years ago

What we've got post #7286 would be:

I'm keeping the upload the artifacts step separate, to allow for us to potentially add complexity there, once TUF comes through.

xavfernandez commented 4 years ago

File a PR, wait for CI, merge on passing CI

and push the tag :)

pradyunsg commented 4 years ago

Oh, I'm pushing the tag after uploading the release. (see last bullet) :P

pradyunsg commented 4 years ago

More things worth automating:

pradyunsg commented 3 years ago

get-pip.py update-and-push

Done in https://github.com/pypa/get-pip/pull/85.

ensurepip update-and-PR

This is external to us, and contingent on CPython's triage/review process which can change. Not worth doing this.

Alright. I'm gonna say that our release process is sufficiently automated. If folks want more automation, file a new issue! :)