search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.63k stars 3.56k forks source link

[CI][Python] Investigate trusted publishing for uploading wheels to PyPI #44733

Open pitrou opened 1 week ago

pitrou commented 1 week ago

Describe the enhancement requested

"Trusted publishing" may help improve the security of wheel uploads, and also provide automated digital attestations. See https://discuss.python.org/t/pypi-now-supports-digital-attestations/71158 and https://trailofbits.github.io/are-we-pep740-yet/ .

It might make it either easier or harder to publish releases, however :)

Component(s)

Continuous Integration, Packaging, Python

pitrou commented 1 week ago

cc @raulcd @assignUser

assignUser commented 1 week ago

We talked about trusted publishing and related GitHub features just yesterday at the infra roundtable!

pitrou commented 1 week ago

Ah, it's nice that you attend those. Thanks a lot :)

raulcd commented 1 week ago

I am curious on how the ASF release process wants to handle those. Currently if we follow the current release process we would be generating the binaries, uploading them to dist.apache.org, verifying them and once the vote passes, downloading them locally from dist and uploading them to PyPI. If we still want to generate binaries and "vote" them, we still would have to upload them to dist.apache.org and after the vote passes create a GH action that downloads the "voted" binaries and uploads those to PyPI, while generating the short lived tokens for "Trusted publishing" this would remove the possibility of an ill intentioned PMC or committer with rights to upload wheels from uploading wrong wheels but there is still the case of someone uploading wrong wheels to dist.apache.org.

A different solution could be to just generate the final wheels once the source has been voted and released and upload those newly generated wheels on the same job that generates them without possibility of tampering (bear in mind source code is signed and could be validated that no modification has happened). I am curious on what other projects do, are other projects not voting on wheels and just publishing afterwards or are they downloading the generated wheels and uploading them as we do?

raulcd commented 1 week ago

cc @kou for feedback too

pitrou commented 1 week ago

I've often said that I don't find our voting process useful. If the binaries are thoroughly verified on CI jobs, then it does not really make sense to have them verified manually, either (especially as the manual verification just consists in executing a script).

kou commented 1 week ago

RubyGems also has "Trusted Publishing" mechanism: https://guides.rubygems.org/trusted-publishing/

If we can use it, I want to use it for RubyGems too.

BTW, could you share what the INFRA said?