search

pyOpenSci / pyosMeta

A package that updates pyOpenSci contributor and package metadata on our website
BSD 3-Clause "New" or "Revised" License
4 stars 17 forks source link

Sign pypi releases using sigstore #156

Closed blink1073 closed 2 months ago

blink1073 commented 2 months ago

Fixes #146

Uses the example given at packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#signing-the-distribution-packages.

I adapted it for the use case of this repository. The major difference is that this repository starts the release process by creating a GitHub Release, while the referenced example creates the GitHub Release as part of the workflow.

The new job runs after the publish job has completed, to ensure that the files have been correctly uploaded to PyPI. It is run in a dedicated signature environment, so that it can have its own set of environment permissions and secrets, if desired. The job is only run against the source repository, not on forks.

The job downloads the distribution files that were created in the build job, and then uses the sigstore/gh-action-sigstore-python action to create signature files for the distribution files. This action requires a GitHub-provided OIDC token, which is why the job has id-token: write permissions.

Finally, the job uploads the distribution and signature files to the GitHub Release that triggered the workflow. This is only done if the workflow was trigged by a release event. This action requires contents:write permission to be able to upload the files to the GitHub release.

I also added a new pre-commit hook that validates GitHub workflows against the published JSON schema.

blink1073 commented 2 months ago

I'm happy to change it to use the same pypi environment as the publish job if desired, otherwise someone with admin access will need to create the new environment. :smile:

lwasser commented 2 months ago

@blink1073 thank you so much for this pr!! i'll will have a look at it this week! i am a bit behind :)

lwasser commented 2 months ago

@all-contributors please add @blink1073 for code, review

allcontributors[bot] commented 2 months ago

@lwasser

I've put up a pull request to add @blink1073! :tada:

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 59.27%. Comparing base (c730c5c) to head (7a12ed7). Report is 4 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #156 +/- ## ======================================= Coverage 59.27% 59.27% ======================================= Files 8 8 Lines 523 523 Branches 83 83 ======================================= Hits 310 310 Misses 208 208 Partials 5 5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

lwasser commented 2 months ago

I'm happy to change it to use the same pypi environment as the publish job if desired, otherwise someone with admin access will need to create the new environment. 😄

@blink1073 can you please help me understand this? I could create an environment OR give you access!

the environment piece within a CI action was new to me here - essentially @webknjaz pointed out that we weren't using best security practices in publishing to pypi! But i don't fully understand where the environment would live outside of telling the action to create an environment and then specifying that environment on pypi. many thanks.

webknjaz commented 2 months ago

@lwasser GitHub Environments is a way to represent deployment targets within GH UI/ecosystem. It's tied to the Deployments API under the hood. When you use a non-existing GH Environment in GHA, it's autocreated. It can also be created manually. Once it exists, you can navigate to the repository settings and configure protections like required reviews, a cooldown timer etc. When set, this is able to pause a GHA workflow right before starting a job marked with an env — a manual button click on the UI would be needed to let it start, giving you more control over letting immutable changes happen on PyPI, for example. If you set the url: key, it also displays a nice link in the workflow graph, that URL can even interpolate variables so you could construct a precise versioned string. The deployments also have a separate web page in the repo. When using trusted publishing, GitHub will present the GH env name as a part of signed piece of metadata that's being sent to PyPI. And so PyPI is able to know and trust that an env with a specific name was used when publishing is requested. You can reconfigure trust on the PyPI side to only allow uploads from jobs where a specific GH env name is set. That's what I suggest doing. This is a part of my guide at https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ by default.

webknjaz commented 2 months ago

@lwasser I didn't look through the PR before writing the first comment. Now that I've done so, I think your use of envs is good. The new env might be something to configure optionally, but it'll work even without that.

lwasser commented 2 months ago

ok so @webknjaz thank you! i didn't fully understand that the environment was autocreated when you run the action.

Screenshot 2024-06-05 at 3 13 17 PM

But i see them now in the settings. so that is why @blink1073 was talking about admin rights. So the new build environment is also there in GitHub.

It sounds like you are saying that it is ok to have both the pypi environment (which is used in PyPI.org to allow the trusted publisher workflow to run) and the build environment that steven added to support sigstore publishing.

Thank you. there is a lot to understand here!!

blink1073 commented 2 months ago

Thanks for the review both of you! I updated the job to use the pypi environment.

webknjaz commented 2 months ago

Thanks for the review both of you! I updated the job to use the pypi environment.

I don't think that should be the same, since this is putting artifacts into a deployment target that is not PyPI. If you want to declare an environment, I'd call it gh-releases or sigstore, maybe. However, it's probably not necessary to mark the job with an environment. If pypi is configured to pause before starting jobs, it will pause before each, but the signing job only runs after publishing, so pausing it there will already prevent it from running until publishing is approved. And if publishing happened, there's no reason to prevent signing that follows it.

webknjaz commented 2 months ago

It sounds like you are saying that it is ok to have both the pypi environment (which is used in PyPI.org to allow the trusted publisher workflow to run) and the build environment that steven added to support sigstore publishing.

It would only matter if you were to do something that is configured within an environment. Like, if you were to have an environment-scoped secret (as opposed to a repo-global one) for some reason, I'd recommend using it. But as things stand, it's probably overengineering, and I'd just remove environment: from this job.

blink1073 commented 2 months ago

But as things stand, it's probably overengineering, and I'd just remove environment: from this job.

Yeah that's fair, done.

lwasser commented 2 months ago

ok thank you both!! i'm going to merge this and then i'll test a small patch release to see how it goes 🚀