search

pytest-dev / meta

Used for generic pytest organization issues, stuff that can impact multiple projects.
https://docs.pytest.org/en/stable/contributing.html#
MIT License
2 stars 1 forks source link

Shared pytest-dev PyPI account/token? #4

Open The-Compiler opened 4 years ago

The-Compiler commented 4 years ago

To push releases, @nicoddemus added a PyPI token with 1d3f27cef076df028ef6434b2d3bd29358c421c3 (which is stored in the PYPI_TOKEN secret in this repo).

Is this your personal account, @nicoddemus? Wouldn't it make sense to create a pytest-dev PyPI user, and then have a token which we can configure as organisation-scoped token for the pytest-dev GitHub organisation? That way, all pytest-dev plugins could add the pytest-dev user to PyPI (which would then be the recommendation instead of "We recommend that each plugin has at least three people who have the right to release to PyPI.").

This way, it'd also be easier for repositories to set up automated deployment via GitHub Actions (which could be another recommendation with an example), as they can use the existing organization-wide token and pytest-dev account instead of using their personal account.

Thoughts?

RonnyPfannschmidt commented 4 years ago

Do the tokens isolate in the sense that plugin repos may not upload pytest releases, i don't want to be paranoid but I'd strongly prefer per project tokens scoped to the pypi distribution in question simply to prevent certain scenarios from larger impact

The-Compiler commented 4 years ago

Nope. I can see where you're coming from, but if someone got hold of your PyPI account (or mine, or Bruno's, or ...) there's probably a lot of damage to be done as well. I can see how the risk would be bigger for a token which can easily be accessed by anyone in the pytest-dev organization, though.

Zac-HD commented 4 years ago

Can we also require that everyone use a hardware token like a Yubikey for 2FA on PyPI?

If cost would make anyone hesitate I'm sure the PSF would look favorably on a grant application to cover it :slightly_smiling_face:

nicoddemus commented 4 years ago

Is this your personal account, @nicoddemus?

Yep!

Wouldn't it make sense to create a pytest-dev PyPI user, and then have a token which we can configure as organisation-scoped token for the pytest-dev GitHub organisation? That way, all pytest-dev plugins could add the pytest-dev user to PyPI

I like the idea of making that more convenient, at the moment it indeed requires a bit of manual setup to get automatic releases going.

However I'm a bit worried about the security implications if such a token gets compromised: that token supposedly would be able upload any pytest-dev package to PyPI.

but if someone got hold of your PyPI account (or mine, or Bruno's, or ...) there's probably a lot of damage to be done as well.

Definitely, but a single token to access to the entire organization seems more problematic, I think.

which would then be the recommendation instead of "We recommend that each plugin has at least three people who have the right to release to PyPI."

I'm not sure having a single key shared by everyone would improve things much, because while right now the recommendation is "We recommend that each plugin has at least three people who have the right to release to PyPI.", we could update it to "We recommend that each plugin is setup for automatic deployment using a token belonging to one of the maintainers", and this is not much different than "We recommend that each plugin is setup for automatic deployment using the token PYTEST_PYPI_TOKEN shared by the organization".

Also, I don't think PyPI has a notion of "organization", where projects could share a token there: new projects joining pytest-dev would still need to give release rights to the owner of PYTEST_PYPI_TOKEN on PyPI.

Does anybody knows how other organizations use these shared secrets? I would be interested in see how that's used, and if it is used for releases.

If cost would make anyone hesitate I'm sure the PSF would look favorably on a grant application to cover it 🙂

If this is a direction we want to take, money shouldn't be a problem because we have funds on OpenCollective that could cover this. 😁

Zac-HD commented 4 years ago

right now the recommendation is "We recommend that each plugin has at least three people who have the right to release to PyPI.", we could update it to "We recommend that each plugin is setup for automatic deployment using a token belonging to one of the maintainers"

I like this as a development workflow, but it can still have problems if a maintainer vanishes.

nicoddemus commented 4 years ago

I like this as a development workflow, but it can still have problems if a maintainer vanishes.

Actually, in this setup, every developer which can push to the repository can make a release even if the owner vanishes: the release is triggered by a tag (pushed by anyone), using that maintainer's token.

Zac-HD commented 4 years ago

And if anything happens to the token - which might be invalidated following a PyPI incident, for example - the project can no longer make releases.

(in practice I think the pytest-dev core team is sufficiently well known to and trusted by the PSF and PyPA team that we could manually recover, but I'd rather model best practices for the rest of the ecosystem than rely on special treatment)

ssbarnea commented 4 years ago

The token itself should not be shared as it is a temporary secret. It is only picked from pypi and saved on github secrets. There are two sets needed, one for production and one of test.pypi.org

Still few org admins need to have control over the credentials of the bot account. Best way is to use a private mailing list as the pypi user address, so credentials can be reset. Be sure you record a nice gravatar for that email address.

There is no standard medium used by others, whatever the admins agreed on (sic). I know teams even using google docs for that (no comment).

Github have decent security measures for preventing secrets exposure in its pipelines. Keep in mind that pypi service account must be only allowed to publish, "maintainer", not admin.

Also setting a secret at org level does not force everyone to use it, you can still define overrides for specific repo if you want. It is a convenience for reducing maintenance over lots of repos, nobody likes have to update 100 repos if tokens need to be refreshed.

That is used by orgs with more sensitive stuff than pytest and I would say publishing using personal accounts pose a bigger risk than a service one (is more likely for one of personal secrets to leaked than the service account).

Keep also in mind that nobody is forcing projects under pytest-dev org to use the organization pypi service account, it is up to them to give it access and enable the publish pipeline, nothing is implicit. Still, I would be more than happy to see it adopted.

The-Compiler commented 4 years ago

Github have decent security measures for preventing secrets exposure in its pipelines.

Accidental exposure, yes. Deliberate exposure can't be prevented. In this scenario, if the GitHub account of anyone being able to push/merge code to master for any pytest-dev plugin gets compromised, it means that person will also be able to release malicious versions of any other plugin or pytest itself.

To be honest, I didn't think of that when opening this issue and I don't like the idea. Thus, I'm -0.5 on this myself.

webknjaz commented 4 years ago

I think that the best thing to have is a PyPI bot account with "maintainer" level of access to the projects. And then, have an API token scoped per project. I also considered having an org-wide secret but then understood that the security implications are unfortunate. I wouldn't even do this for TestPyPI.

graingert commented 4 years ago

Best way is to use a private mailing list as the pypi user address, so credentials can be reset.

One way is to store a private key in github actions and encrypt api keys with that key in git using password-store (pass) then anyone can roll the key or add new keys travis style

webknjaz commented 4 years ago

One way is to store a private key in github actions and encrypt api keys with that key in git using password-store (pass) then anyone can roll the key or add new keys travis style

Or use Keybase and mount it as fuse wherever necessary

ssbarnea commented 4 years ago

When the ticket was opened I was ready to bet some over engineered solution would be proposed that would likely negate all the other benefits... I hope we would evaluate the risk and possible damage before making a decision. There is no perfect security. I would also propose experimenting with less mainstream changes and plugins and adapt based on what we learn when we scale.

RonnyPfannschmidt commented 4 years ago

personally i'd prefer that a few of the core maintainers just set up the project specific tokens as part of the organization maintenance/take-in, and we have a pytest-dev-upoader user on pypi that would be maintainer on all the projects

a solution for sharing the login credential by the core maintainers has to be found tho (pass sounds like something i'd like for that)

in essence

webknjaz commented 4 years ago

@RonnyPfannschmidt

4) it'd be cool to also have some sort of a linter to ensure that this bot user does not have "owner" access to those packages, only "maintainer"

The-Compiler commented 4 years ago

@RonnyPfannschmidt If you have the same user anyways, what's the benefit of having separate tokens (other than being able to revoke them more easily)? The permission issue you mentioned above still stands then.

RonnyPfannschmidt commented 4 years ago

@The-Compiler this is primarily to prevent plugin maintainers under pytest-dev to be targets for having their github accounts be used to upload releases of other pytest plugins and/or pytest itself by hijacking one of the other repositories

in particular for the plugins with lower traffic its thinkable that a actual attack could take a little while to be detected, i understand the scenario is rather unlikely but i really want to err on the side of what would weight on the conscience of the maintainer who got used in such a attack as well as i want to avoid making the maintainers of other plugins a great attack target for core pytest or important pytest plugins

The-Compiler commented 4 years ago

Ah, I forgot that it's possible to scope an API token to a specific project even if they're all using the same user. Yeah, that would seem like a nice solution then: One account, but still some kind of privilege separation between the different tokens/projects.

BeyondEvil commented 4 years ago

personally i'd prefer that a few of the core maintainers just set up the project specific tokens as part of the organization maintenance/take-in, and we have a pytest-dev-upoader user on pypi that would be maintainer on all the projects

a solution for sharing the login credential by the core maintainers has to be found tho (pass sounds like something i'd like for that)

in essence

  • each project/repo gets a own token
  • tokens belong to a pytest-dev-uploader user (to be created/exact name to be chosen)
  • that users should have 2factor auth and core maintainers should have a way to access it

This is how we've solved it at my current place of work w.r.t. to DevOps stuff. We have one bot, but each project/repo/whatever has its own token (where possible).

graingert commented 4 years ago

It might be worth looking at the https://jazzband.co/about/releases approach

On Fri, Jul 31, 2020, 14:05 Jim Brännlund notifications@github.com wrote:

personally i'd prefer that a few of the core maintainers just set up the project specific tokens as part of the organization maintenance/take-in, and we have a pytest-dev-upoader user on pypi that would be maintainer on all the projects

a solution for sharing the login credential by the core maintainers has to be found tho (pass sounds like something i'd like for that)

in essence

  • each project/repo gets a own token
  • tokens belong to a pytest-dev-uploader user (to be created/exact name to be chosen)
  • that users should have 2factor auth and core maintainers should have a way to access it

This is how we've solved it at my current place of work w.r.t. to DevOps stuff. We have one bot, but each project/repo/whatever has its own token (where possible).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pytest-dev/pytest/issues/7585#issuecomment-667109449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADFATHU3CMSVIZSCSFYJI3R6K6RBANCNFSM4PNTINTA .

symonk commented 4 years ago

Given pytest core is included in circa 200k~ repositories (if github is accurate) we should tread with caution here :) Keeping access as narrow as possible to function effeciently (imo).

bluetech commented 4 years ago

FYI pytest-django is stuck in token limbo. It does have automatic deployment but the token must have been revoked. I probably made a mistake or overstepped in trying to release without getting an OK from the maintainers, but having the project under pytest-dev was meant for such situations IMO. I don't think there is a technical solution to such problems but it does show the risks of relying on individual tokens.

graingert commented 4 years ago

A probot application might help here?

nicoddemus commented 4 years ago

FYI pytest-django is stuck in token limbo. It does have automatic deployment but the token must have been revoked. I probably made a mistake or overstepped in trying to release without getting an OK from the maintainers, but having the project under pytest-dev was meant for such situations IMO.

Yes, having it under pytest-dev was meant for such situations. I think what happened in pytest-django was not usual. @pelme could you add me and/or @bluetech as publishers on PyPI? We can take it from there.

pelme commented 4 years ago

FYI pytest-django is stuck in token limbo. It does have automatic deployment but the token must have been revoked. I probably made a mistake or overstepped in trying to release without getting an OK from the maintainers, but having the project under pytest-dev was meant for such situations IMO. I don't think there is a technical solution to such problems but it does show the risks of relying on individual tokens.

@bluetech Just for the record - I am very happy that you offered to help getting the release out. Thank you.

I think it would be good if both you and @nicoddemus were PyPI owners to avoid situations like this in the future. Not sure how many people have PyPI on other pytest-dev projects? But feel free to add more people if you think it is a good idea.

ssbarnea commented 4 years ago

I already created a generic bot account https://pypi.org/user/pytest-dev/ and configured it for the two plugins I actively maintain. I can generate project specific tokens for it and I am really interested in sharing access to this account with others. This accounts uses a private google group address, so we avoid depending on a single person. @bluetech @nicoddemus do you want me to send you an invite?

PS. One admin should transfer this ticket to https://github.com/pytest-dev/meta -- same applies to other infra tickets.

nicoddemus commented 4 years ago

I already created a generic bot account pypi.org/user/pytest-dev and configured it for the two plugins I actively maintain. I can generate project specific tokens for it and I am really interested in sharing access to this account with others. This accounts uses a private google group address, so we avoid depending on a single person. @bluetech @nicoddemus do you want me to send you an invite?

I re-read the whole thread to make sure I didn't miss anything, but apologies if I missed a detail or another.

The consensus is having a PyPI bot user, pypi.org/user/pytest-dev. This actually a private mailing that contain just a few people, so everybody is on the loop in case tokens get created/revoked.

When a new project joins the organization:

  1. We will strongly suggest that they setup automatic deployment with GitHub actions, using a specific token name, PYPI_TOKEN.
  2. One of the users that has access (say me) will log to pypi.org/user/pytest-dev using the shared password (and two-factor authentication, but not sure it is possible to have more than one authentication in place per user?)
  3. I create PYPI_TOKEN for that project only, with publish scope.
  4. I go the project settings on GH and paste the new token (as I have admin/maintainer rights there).

So even if the bot account gets compromised (someone gets a hold of its shared password), one attacker can do revoke the tokens and cause a bit of havoc by deleting packages. But real power to cause problems come from getting hold of the actual GitHub credentials of a pytest-dev maintainer.

Suppose we have 4 people with access to the bot account. IIUC, that would be the same as suggesting that all same 4 people be added as publishers on PyPI for pytest-dev projects, except that we have a central point of communication to coordinate when needed (also people come and go of the project). Is my understanding correct?

If so, then I think this is a good idea in the end, and thanks a lot for everyone to join in the discussion.

I have a few other questions:

  1. A private group email is not very secure, perhaps another channel to the exchange of secure information is necessary. Keybase seems to have gone out of favor with many, perhaps we can use another secure channel to exchange secure information? DeltaChat comes to mind due to its encryption and distributed nature, and also all you need is an e-mail.

  2. We probably need to have a plan in case the share account is compromised somehow.

Thoughts?

ssbarnea commented 4 years ago

One note: the bot is only a pypi package maintainer, not an owner. Maintainers can only make new releases, so a potentially compromised bot would only make a new release for a specific package only.

Regarding shard account compromise risk: keep list of owners small and assure they have a direct communication channel. Even if that account gets hacked (should never), pypi admin can step in and fix it, they do stuff like this even.

nicoddemus commented 4 years ago

Maintainers can only make new releases, so a potentially compromised bot would only make a new release for a specific package only.

Just to be clear: the compromised bot is shared by all pytest-dev projects, so it should be able to make releases for all pytest-dev projects.

We still need to decide on how to share the password to move this forward.

The-Compiler commented 4 years ago

We still need to decide on how to share the password to move this forward.

Potential ways to do that:

nicoddemus commented 4 years ago

How about DeltaChat. Seems like a good option due to being free, distributed in nature, and having strong encryption.

The-Compiler commented 4 years ago

That works great to share things one-time, but over the years that seems like quite a pain. If people come and go, you'll need to find out whom to contact and people will need to search for old messages (unless they added passwords to their own password manager) rather than having a well-defined place to look at.

nicoddemus commented 4 years ago

Good points, thanks! 👍

ssbarnea commented 4 years ago

Afaik, telegram also offers end to end encryption, so is an option for the very uncommon credential sharing action.

On Thu, 24 Sep 2020 at 16:12, Bruno Oliveira notifications@github.com wrote:

Good points, thanks! 👍

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pytest-dev/meta/issues/4#issuecomment-698408198, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZAX7QX2QRU4HNF4B2Y6LSHNOVRANCNFSM4RLJJZ2Q .

--

/sorin

RonnyPfannschmidt commented 4 years ago

I would suggest something like a private password store repo with all the credentials for pytest related infrastructure