Proposal: preliminary support for PEP 740

[woodruffw]

Is there an existing issue for this?

• [X] I have searched the existing issues (open and closed), and could not find an existing issue

What keywords did you use to search existing issues?

signing, provenance, PEP 740

Please describe the problem you are attempting to solve with this request

This is a proposal to add preliminary support for PEP 740 to twine. I (and others at Trail of Bits) will perform all of the engineering necessary on twine if this proposal sounds good.

Context:

PEP 740 is the intended replacement for PyPI's (now removed) GPG signature support. It's currently in the final stages of the PEP process, and we're beginning to plan its rollout (with the first phase building on PyPI's support for GitHub Actions as a trusted publisher). As part of that, we're looking at twine integration as part of easing adoption within gh-action-pypi-publish.

How do you think we should solve this?

I would like to add a preliminary version of PEP 740 support to twine. As a rough design sketch:

1. twine upload learns the --attestations (or --experimental-attestations) flag
2. When --attestations is passed, twine upload discovers adjacent {dist}.*.attestation files for each {dist} (or if the user performs twine upload dist/* or similar, twine upload performs matching/disambiguation like it currently does for .asc)
3. Each attestation is loaded and added to its file's upload via the new attestations form field, per PEP 740's upload endpoint changes
4. The upload process continues as normal

Without --attestations, twine will retain its current behavior (of not being aware of adjacent .attestation files and thus treating them as separate inputs). This ensures (1) backwards compatibility, and (2) that the default does not change while we experiment with the implementation here.

Under this proposal, twine itself is not (and will not be) responsible for producing the attestations themselves: the attestations are produced by the CI/CD and/or build process. This could be changed in the future if there's interest, but keeping attestation generation separate from twine should keep maintenance burden for this feature lower.

Anything else you'd like to mention?

As mentioned above: if this proposal (or an adaptation) sounds good, my team and I will do 100% of the work on twine here! We wanted to file this issue first to offer some awareness/visibility before springing any proposed changes on you, especially since the current proposal includes a CLI change 🙂

There is also an ongoing discussion thread for PEP 740 on DPO: https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498

CC @di @sethmlarson @webknjaz @facutuesca for visiblity

[sethmlarson]

Thanks @woodruffw!! This is exciting to see. Wanted to clarify that today we'd be uploading an attestation about publish provenance per-artifact, but in the future we're likely to have build provenance provided by a builder workflow as well. Wanted to make sure you're considering this future use-case so we don't have to step on toes in the future :)

[woodruffw]

Wanted to make sure you're considering this future use-case so we don't have to step on toes in the future :)

Thanks for pointing this out! This was indeed an oversight on my part 😅 -- to support multiple attestations per-file the .attestation input will really need to be .attestations (i.e. a JSON array of each attestation), per PEP 740's language. I'll update the top-level comment to reflect that!

Edit: @sethmlarson pointed out over chat that having this be .attestations means that somewhere in the pipeline JSON array merging needs to happen. So I'm proposing this as {dist}.*.attestation instead, with the idea that twine upload --attestations will collect everything matching that pattern for {dist}.

[sigmavirus24]

This sounds great. I'd prefer smaller PRs personally because I have very limited time outside of work and that doesn't look like it will improve for several months. Anything I can review easily from my phone will be helpful

[woodruffw]

I think we're pretty much done here! All of the required preliminary features have been merged, and I'll keep an eye out for bugfixes/improvements as we build up the PyPI and actions sides 🙂

@sigmavirus24 Would you be willing to do a pre-release or minor release for the changes so far? Per semver this would be 5.1.0 I think, but a pre-release would also help us integrate this for testing if a stable release isn't acceptable to you 🙂

[woodruffw]

(If a release is okay, I'm happy to send a PR updating the versions and prepping it to whatever extent is convenient for you.)

[sigmavirus24]

Sounds great to me @woodruffw

[woodruffw]

Thanks @sigmavirus24! I used your release docs (thanks for those) to open #1107 🙂