search

scientific-python / upload-nightly-action

This action is used to upload nightly builds of your package.
https://anaconda.org/scientific-python-nightly-wheels
BSD 3-Clause "New" or "Revised" License
7 stars 10 forks source link

NumPy tokens have expired #90

Closed mattip closed 1 month ago

mattip commented 1 month ago

I am not sure who are the numpy project admins. The token has expired, as can be seen in this workflow and this workflow cc numpy/numpy#27339. There are two tokens: NUMPY_STAGING_UPLOAD_TOKEN and NUMPY_NIGHTLY_UPLOAD_TOKEN.

matthewfeickert commented 1 month ago

@mattip This is a NumPy issue, not a upload-nightly-action issue. Please ask @rgommers @tylerjereddy and @seberg to make new tokens via https://anaconda.org/<anaconda cloud user name>/settings/access.

c.f.

https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/README.md?plain=1#L39-L45

matthewfeickert commented 1 month ago

Though you can see by looking at https://anaconda.org/scientific-python-nightly-wheels/numpy/files

image

that @seberg's token was being used for the uploads. So if you want that to stay the same, @seberg please regenerate the tokens and update the relevant NumPy GitHub Actions secrets.

mattip commented 1 month ago

Please ask ...

I think you just did :)

matthewfeickert commented 1 month ago

I think you just did :)

Indeed. :) Though you might have more direct channels than I do, so if so, and this is time sensitive please convey the information there also.

tylerjereddy commented 1 month ago

@mattip is it urgent or can it wait for i.e., Sebastian on Monday?

mattip commented 1 month ago

Not urgent, thanks.

stefanv commented 1 month ago

Do we have a way of checking who has access to renew tokens? I have permission to set tokens on gh/numpy, but not generate them on anaconda.org.

seberg commented 1 month ago

Sorry, I created a new token now. I think we set up a token for NumPy with my account but never added a second admin with access to the NumPy nightlies upload. (Which doesn't seem like a huge deal, since I think adding a second person is easy, if I was MIA for longer, but also happy to just add e.g. Matti.)

matthewfeickert commented 1 month ago

@rgommers @tylerjereddy and @seberg all have admin privileges for the nightlies channel. We can add whoever else you would like and just need their Anaconda Cloud username.

seberg commented 1 month ago

Ah, didn't realize that we already had 3 people, that seems plenty. Anyway, just added a token, it'll expire in a year+some (maybe should have done longer), and updated it in NumPy repo, plus the PR for Cirrus.

I think we didn't notice the upload was failing for a while, which seems like the bigger annoyance then actually (it's OK if they fail for 1-2 weeks or so IMO, but not nice to miss for a month)

stefanv commented 1 month ago

On https://github.com/notifications, I've set up a custom filter with reason:ci-activity. This shows me all failed runs. Not perfect, but it's something.

matthewfeickert commented 1 month ago

Do we have a way of checking who has access to renew tokens? I have permission to set tokens on gh/numpy, but not generate them on anaconda.org.

@stefanv (sorry was answering on phone earlier so missed this)

(I might be telling you things you already know but) people are given admin privileges to individual project groups (https://anaconda.org/scientific-python-nightly-wheels/groups) that allow tokens generated by them to have upload access, but they generate the tokens on their personal Anaconda Cloud accounts

https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/README.md?plain=1#L39-L45

(The people that have the ability to generate tokens for the scientific-python-nightly-wheels Anaconda Cloud org are in the Owners group — which includes you — but those tokens are for admin reasons and aren't/should not be used for uploading nightly wheels from any projects.)

matthewfeickert commented 1 month ago

As I see that the uploads for https://anaconda.org/scientific-python-nightly-wheels/numpy/files are working again with @seberg's token I'm going to close this issue.

image

Though if people have additional comments or questions please ask them here so that we can clarify things and/or create new issues to improve the process.

mattip commented 1 month ago

Is there a reason to have a 1 year expiration time for the token?

tupui commented 1 month ago

Security. 1 year is already extremely generous for what this token can do in the wrong hands.

stefanv commented 1 month ago

It's quite painful to rotate tokens, though. It would be helpful if we had a script that a maintainer could run, that would e.g. connect via OAuth, to generate tokens and set them where necessary. This would be a good tool for SP to provide, if gh can't do it already.

stefanv commented 1 month ago

(According to SPEC 6, these procedures, and who have access to them, should be documented. But not much progress on that front yet.)

charris commented 1 month ago

I think we didn't notice the upload was failing for a while

I got messages that the emscripten upload failed and opened an issue the second time it happened. I don't know why none of the other failed uploads sent me a message.

EDIT: Hmm, emscripten uses an older procedure for the upload, that probably explains it. It seems that the upload procedure should ping someone on failure, maybe that can be configured.

tupui commented 1 month ago

Yeah I've seen these attempts at a few companies now and the best solution I've seen, as in the thing which team can rely on in the long run, is an entry in a calendar 😅 There are plenty of tools to help with that too, just yet another thing to have configured and maintain E.g. https://shopify.engineering/automatically-rotate-github-tokens

bsipocz commented 1 month ago

I got messages that the emscripten upload failed and opened an issue the second time it happened. I don't know why none of the other failed uploads sent me a message.

Unfortunately, failing cron CI statuses in GHA are not consistent to send notifications.

matthewfeickert commented 1 month ago

It seems that the upload procedure should ping someone on failure, maybe that can be configured.

@charris Failing GitHub Actions jobs already send emails (to whoever it thinks was responsible for triggering the schedule GitHub Actions job, which apparently is (c.f. https://github.com/orgs/community/discussions/47855#discussioncomment-7061183) the last person to edit the cron syntax in the GitHub Actions workflow file, and so in the case of the linked failing two workflows in the original issue post you and @seberg). What sort of additional notifications that something went wrong were you thinking about?

edit:

emscripten uses an older procedure for the upload, that probably explains it.

The emscripten upload to the https://anaconda.org/scientific-python-nightly-wheels channel currently uses the v0.5.0 release of the scientific-python/upload-nightly-action GitHub action

https://github.com/numpy/numpy/blob/f3b3dc536e624b20822cd37cd01c15e3767af84a/.github/workflows/emscripten.yml#L80-L84

so I don't follow what "older" means here. Can you give a bit more context on what you mean?