search

pypi / warehouse

The Python Package Index
https://pypi.org
Apache License 2.0
3.59k stars 965 forks source link

Add support for two-factor authentication #996

Closed edmorley closed 5 years ago

edmorley commented 8 years ago

Account compromise of the owner of a popular package on PyPI is something that would have pretty dire consequences. 2FA/MFA would be a big step to helping avoid this.

This would need to be implemented in both:

The 1st of these is presumably the easiest, and I believe will still add some value even if the 2nd isn't implemented initially - since it will still prevent things like package owner/maintainer role changes or modification of a user's listed GPG key or login details (presuming these cannot be made via the API).

Harder is what to do with the API, given it requires client changes and is also used by some in a non-interactive manner in automation. I see a few options:

  1. Add 2FA support to setuptools/pip, and just document that people uploading packages in automation will just need to disable 2FA for their whole account (or ideally create a separate account just for automation).
  2. Add support for API keys to both Warehouse and clients (#994) and allow those to circumvent 2FA. For the people that really must upload packages non-interactively this would still protect against a few risks (credentials re-use, weak credentials), and the API keys could also be made package-specific or tied to a particular IP range. For everyone else, they would just not create an API key and still have full 2FA protection.
  3. All of the above.

In terms of implementation, there are a number of packages that support TOTP (used by Google Authenticator): https://pypi.python.org/pypi/onetimepass/1.0.1 https://pypi.python.org/pypi/pyotp https://pypi.python.org/pypi/oath https://pypi.python.org/pypi/otpauth

(I've filed this against warehouse since I'm presuming this is beyond the scope of maintenance-only changes being made to the old PyPI codebase)

dstufft commented 8 years ago

I do plan on adding 2FA to Warehouse, though it will be a post-launch task and will likely require deprecating the current upload API (which I want to do anyways). As far as what technology we use to handle it, TOTP is an obvious shoe-in. I'd like to explore some options for services that handle 2FA authentication though and see if any of them would donate service (assuming the export story to allow us to go away from their service in the future is sane). I haven't done a ton of exploring in this idea yet, but it would be nice if we could support more than just TOTP (particularly, something like Authy's OneTouch would be really nice).

ryanhiebert commented 8 years ago

FWIW, I've been impressed with U2F, which is supported by GitHub and Google currently, of the sites I use regularly. Built-in support for Chrome, and an add-on to support it in Firefox, and MS has registered their intent to implement it as well.

TOTP is still good, since it's much more available, but I'd love to see U2F supported as well.

edmorley commented 8 years ago

Yeah U2F would be great too, I meant to mention that in the OP. Thank you for your replies :-)

lukesneeringer commented 6 years ago

Note: I am planning on taking on #994 and this (in that order). See more discussion in #994.

brainwane commented 6 years ago

Thanks, @lukesneeringer!

Since the legacy site doesn't have this feature, I'm marking this issue for a milestone further along, so we don't block the shutdown of the legacy site while waiting for this feature.

mschwager commented 6 years ago

Hi all,

I'm happy to see that Warehouse is planning to add 2FA!

I'd like to suggest adding support for Duo 2FA (full-disclosure I'm an engineer at Duo). Duo supports many of the technologies mentioned here:

Duo also has a free edition that developers using the Warehouse could use to protect their logins for free! Adding Duo 2FA support is as simple as following the Duo Web SDK docs and using our Duo Python SDK.

It'd be great if multiple 2FA solutions could be used. E.g. have a generic TOTP solution, OR Duo 2FA, OR ... and it's as easy as implementing an interface. I'd be happy to contribute code to make this happen.

P.S. The new Warehouse is awesome! Great work!

brainwane commented 6 years ago

Thanks, @mschwager! Will you be PyCon North America later this month, where we'll especially discuss this during the sprints May 14-17? https://wiki.python.org/psf/PackagingSprints

mschwager commented 6 years ago

Hey @brainwane,

Unfortunately, I will not be at PyCon... unless you've got a spare ticket ;)

I'd be happy to continue discussion on this issue, though, if someone wouldn't mind transcribing!

brainwane commented 6 years ago

@mschwager I am 98% sure that sprints are free to attend even if you did not get a ticket to register for the main conference! If you can come to Cleveland, please do!

brainwane commented 6 years ago

I'm right -- see https://us.pycon.org/2018/community/sprints/ and https://twitter.com/pycon/status/993923892002607104 . Can you make it?

takluyver commented 6 years ago

Heads up, a user on reddit found a package on PyPI with a very obvious backdoor to steal SSH keys. It looks like someone other than the author may have got his PyPI credentials and uploaded it without his knoweldge, although the details are not very clear yet.

https://www.reddit.com/r/Python/comments/8hvzja/backdoor_in_sshdecorator_package/ https://github.com/urigoren/ssh_decorator/issues/11

di commented 6 years ago

Thanks @takluyver, we are aware. For a number of reasons, it's highly unlikely that this user's account was truly compromised.

steiza commented 6 years ago

Hey @brainwane! I work with @mschwager, will be at PyCon, am staying for the sprints, and would love to talk about improving Python package security. I'll look for the BoF / Open Space as a starting point?

tomfbiz commented 6 years ago

I am at Pycon North America, and am starting to do some research on this ticked (compare packages listed for TOTP, look at where the code will need to be changed)

di commented 6 years ago

From IRC: Some notes on the subject are here: https://etherpad.wikimedia.org/p/PyPA_Warehouse_2FA_notes

tomfbiz commented 6 years ago

Notes from spending a morning at PyCon North America Sprint researching and thinking:

Simple evaluation of TOTP packages:

https://pypi.org/project/onetimepass/1.0.1/. 483 stars not updated since 2015 https://pypi.org/project/pyotp/ 853 stars updated in 2017 & 2018 https://pypi.org/project/oath/ 104 stars Oauth not just TOTP, last 2 commits in 2016 and 2015 https://pypi.org/project/otpauth/ 87 stars 2 updates in 2018, then 2015

Based on this simple evaluation, pyotp looks like the best to start with https://github.com/pyotp/pyotp

TOTP description from from wikipedia : https://en.wikipedia.org/wiki/Time-based_One-time_Password_algorithm

In a typical two-factor authentication application, setup proceeds as follows: a user enters username and password into a website or other server, the server generates a secret key which the user enters on to their TOTP application on a smartphone or other device (often by scanning a QR code). To verify that process worked, the user application immediately generates a one-time password to be checked by the server. On subsequent authentications, the user enters their username, password and the current one-time password. The server checks the username and password as normal then also runs TOTP to verify the entered one-time password. For this to work, the clocks of the user's device and the server need to be roughly synchronized (the server will typically accept one-time passwords generated from timestamps that differ by ±1 time interval from the client's timestamp)

QR code generation Need to generate a QR code to scan in the Google Authenitcate app. https://github.com/neocotic/qrious is "Pure JavaScript library for QR code generation using canvas. Here is some code to generate a QR code in pyramid Image must not be stored on disk https://stackoverflow.com/questions/38525722/generate-a-qr-code-in-pyramid-view-return-it-without-writing-to-disk#answer-38529362

Probably need a way to display the token as a string for users to save for backup as well.

Functional requirement decisions

Do we want a bypass if someone loses their device, like some sites provide?

Do we want to do 2fa for each login, or allow users to 'remember' a device with a cookie and only use 2fa for new devices? Do we want to allow SMS as well as TOTP? This will depend on a 3rd party to send the SMS (Twillow is an example of that) There may not be a free option for this. (NIST recommends against using SMS as a second factor) Another are to think about: how do we support 2FA for package upload (e.g. how would 2FA work with twine?)

Another thing to consider: are we only supporting TOTP or are we also supporting U2F devices (e.g. yubikey)?

Blog Tutorial examples Here is a tutorial with an example implementing TOTP (as well as SMS using Twillow) https://www.twilio.com/blog/2013/04/add-two-factor-authentication-to-your-website-with-google-authenticator-and-twilio-sms.html While this example does not use libraries, it is a good sample reference for the flow need. It does not show where a user will get the plaintext secret to archive for backup, which seems like a good practice. The SMS is at the end of the tutorial after the TOTO part. Here is another example.tutorial using onetimepass https://blog.miguelgrinberg.com/post/two-factor-authentication-with-flask

Proposed UI changes for Warehouse Currently, register redirects to homepage after registering. Some pages (but not the homepage include a header warning "Warning Your primary email address (tomfbiz@gmail.com) is unverified. Verify your email or add a new address." I propose that we either add a landing page after the user logs in explaining options or send people to the /manage/account/ page after registering with info on that page. After that I suggest we mimic the flow/page structure from the twillow example /verify-tfa page to enter TFA field when logging in /enable-tfa-via-app to set up TFA (with QR code)

If we were to support SMS, there would be 2 more corresponding pages.

Database change(s) Need to add otp_secret string to the User model. Presumably, we could this both to store the secret and to indicate that the user has enabled 2FA -- this will be Null (or empty string) if the user does not have 2FA enabled.

mschwager commented 6 years ago

Just to follow up, looks like y'all have focused on an OTP approach for now? As opposed to a particular vendor?

That seems like a good start, especially if it's built with extensibility in the future in mind :)

ryanhiebert commented 6 years ago

TOTP is good and widely available. I'd love to see U2F implemented with the recent WebAuthn draft standard, it seems like that's going to be the way to integrate with tokens going forward.

takluyver commented 6 years ago

Fallbacks: I think it's worth having the option to generate some time-unlimited one-time codes, which you can keep somewhere to regain access if you lose your device. I have such codes for my Github and Lastpass accounts, for instance. Possibly it could email you when a one-time code is used, so it's harder to do so surreptitiously.

I think admins shouldn't be able to disable 2FA for a user account. If they can, there will inevitably be requests for them to do so, and then how do they verify that you're the real owner? It's better to make it clear that there's no reset if you lose access.

2FA for uploading packages: Github does something similar for API access - I know of some code using it here: https://github.com/minrk/ghpro/blob/1.1.1/ghpro/api.py#L58-L75

To be a bit more ambitious, the computer you're uploading from could be the "something you have" factor, so once you've authorised it to do uploads, you don't need to get your phone out for every upload.

edmorley commented 6 years ago

Do we want to allow SMS as well as TOTP?

Given all of the issues around SMS, I would prefer if it wasn't supported as a fallback option.

mschwager commented 6 years ago

As it turns out, I may have some time to look at this more closely in the coming months, and was hoping to sync up. I wanted to double check that it's not already being worked on, and validate my approach. Would PyPA expect some kind of specification or formalization of work before working on this, or is simply submitting a PR sufficient?

At this point I would be implementing @tomfbiz's general outline above using TOTP + QR and injecting a 2FA page into the authentication workflow. I could elaborate more, but basically I'd be implementing the plan over the coming months.

Thoughts?

Also, what's the best way to communicate around this project? The IRC channel?

dstufft commented 6 years ago

For real time communication, #pypa-dev on Freenode is best, for async communication regarding Warehouse/PyPI specific design decisions, this issue or a linked PR to this issue is best. For any larger questions that require changes across multiple tools (e.g. if we need to teach twine about 2fa) then distutils-sig is the best place for those kinds of discussions (or at the very least, giving distutils-sig a heads up that there is relevant discussion happening here).

As far I'm aware, there's not anyone working on this, so it's certainly available for someone to take this on.

As far as a PR versus a design doc... that really depends! We don't have a formal process for that kind of thing (except when designs have far reaching implications across the entire system, we go to a PEP, but I don't think this would rise to that level). Really whichever way you feel most comfortable with is fine, the biggest risk to sending a PR first is that you might have to iterate on it more to come up with a solution folks are happy with versus spending that effort up front on a design doc (but it might also be less work!).

As far as actual implementation and what we support. I think we should support both TOTP and U2F (although we don't need both in the initial PR, as long as the code doesn't back us into any corners in one direction or another) and I do not believe that we should support SMS.

I am ambivalent between the idea of implementing these ourselves or using a service to implement them.

If we do use a service, then it should follow the basic pattern that we've used throughout the site, that we don't expose proprietary protocols/domains/etc to end users (e.g. they have to be contained entirely in the backend), we need a migration plan/path we can take to another service (or doing it ourselves) if needed (e.g. can we export data?), and finally we'd need the service to either be a free service, or the PSF and that service would need to come to a sponsorship agreement.

Implementing it ourselves makes long term questions about keeping that feature working easier to answer (we own it, so we just maintain it like anything), so there is a slight preference for that. However if there's a reasonable value add and a service that's willing to sponsor and works within our constraints, that's OK too.

As far as specific workflow goes, @nlhkabu is the final arbiter on that-- but I think that the first thing is getting the functionality working, then we can figure out the best UX for exposing it to people.

Does that all make sense?

mschwager commented 6 years ago

Thanks for all the information! It all makes sense and is very helpful. I'll definitely be planning to work on this piece of functionality over the coming months and will keep this issue updated.

In the mean time, I'll go over some of the open questions and leave my thoughts.

ad-m commented 6 years ago

In the last time, some npm packages have been compromised to steal tokens to the npm API (https://blog.npmjs.org/post/175824896885/incident-report-npm-inc-operations-incident-of ). In consequence, npm revoked all API keys and introduced two-factor authentication ( https://blog.npmjs.org/post/175861857230/two-factor-authentication-protection-for-packages ). This can be a valuable experience for us.

sergeykolosov commented 6 years ago

I've got experience doing both backend and UX for TOTP-based 2FA (including recovery flows), and will happily review whichever the artifacts produced within this issue.

@nlhkabu feel free to ping me whenever there's anything UX-related to have a look at.

mschwager commented 6 years ago

Do we want a bypass if someone loses their device, like some sites provide?

Having bypass codes as a backup for lost devices is a typical workflow that I've seen with similar 2FA implementations.

Allow an admin to disable 2fa We could provied an emailed time llimited link that would disable 2FA, similar to a "forgot password link". This lowers the level of security to email, which may be ok, but not ideal. Another solution would be to email a time limited one time code that could be used without an app, which could be entered on the site (or in apps like twine).

If I understand this correctly, here we're considering the case where a device is lost (or user is unable to 2FA for some reason), AND they're unable to use their backup mechanism as well (could be backup codes as mentioned above).

I agree that backup to email is a solution, but I also agree that it's not ideal because we're reducing the security of the authentication to the security of the email account. We almost certainly don't want this situation to fall into the hands of PyPI admins verifying users. Due to the scale of users and the lack of proper rigor in determining a user's identity, this approach would be difficult.

I think this situation warrants further investigation. It'd probably be worthwhile to see what Github, Google, etc do in this situation.

Do we want to do 2fa for each login, or allow users to 'remember' a device with a cookie and only use 2fa for new devices?

As a user I'd expect "Remember me" functionality to fully remember me, not just a portion of the authentication. Although, this could also be a configuration option when setting up 2FA.

Do we want to allow SMS as well as TOTP? This will depend on a 3rd party to send the SMS (Twillow is an example of that) There may not be a free option for this. (NIST recommends against using SMS as a second factor)

I say we start with a minimally viable 2FA implementation, then add more features as we go along. Starting with TOTP then working toward U2F, SMS, Push, etc in the future seems reasonable to me.

Another are to think about: how do we support 2FA for package upload (e.g. how would 2FA work with twine?)

Similar to above, I say we start with 2FA in the web console, then expand from there.

Another thing to consider: are we only supporting TOTP or are we also supporting U2F devices (e.g. yubikey)?

See above.

Thoughts?

brainwane commented 5 years ago

Great news - @woodruffw is working on this right now! See #5567 for his work in progress.

brainwane commented 5 years ago

Wes Turner asked:

Is webauthn the multi-factor / 2FA spec to implement now? It's now approved; so while you experts are working on it it may be worth a look to just implement webauthn while we have funding for experts....

(Further details at distutils-sig.)

woodruffw commented 5 years ago

Is webauthn the multi-factor / 2FA spec to implement now? It's now approved; so while you experts are working on it it may be worth a look to just implement webauthn while we have funding for experts....

Yep! That's my plan. Of course, not all of Warehouse's supported browsers can do Webauthn yet: Firefox, Edge, and recent Chrome (67+) have it, but Safari only has it in technical previews and IE support is nonexistent. This is a while lot better than FIDO/U2F support (which is pretty much just Chrome + Firefox with an about:config setting), but it's still worth keeping in mind. In any case, all users will still be able to add TOTP as a second factor.

mlissner commented 5 years ago

IE support is nonexistent

Has anybody checked their new Chrome + IE frankenstein yet?

woodruffw commented 5 years ago

Has anybody checked their new Chrome + IE frankenstein yet?

It looks like their plan is to build it as a shell over Chromium, so my guess is that they'll get Webauthn support.

brainwane commented 5 years ago

Per #5661 we are going to begin to roll out TOTP-based 2FA this month. This Friday May 3rd we plan to flip the switch so users of test.pypi.org can use 2FA, and we'll soon be announcing where users of pypi.org can sign up for a private beta so they can test 2FA on the main PyPI site May 3rd-20th.

Once we shake out bugs through that, we plan to enable the 2FA feature for all PyPI users, and move on to working on WebAuthn support.

brainwane commented 5 years ago

I'm about to start sending out this announcement pointing to this wiki page about testing 2FA.

brainwane commented 5 years ago

We're reviewing, triaging, and fixing bugs we're finding during the TOTP beta, and @woodruffw is working on WebAuthn support in #5795 (that WIP is good enough for Warehouse developers and PyPI power users to spin up and try now).

webknjaz commented 5 years ago

May I suggest adding smth like &image=https://pypi.org/static/images/logo-small.6eef541e.svg to the TOTP QR code? Some apps like FreeOTP can recognize this param and show it as an icon next to the OTP entry.

webknjaz commented 5 years ago

I've tried out 2FA on test PyPI. Disabling it seems too easy: shouldn't PyPI require entering an OTP to complete this action?

woodruffw commented 5 years ago

May I suggest adding smth like &image=https://pypi.org/static/images/logo-small.6eef541e.svg to the TOTP QR code? Some apps like FreeOTP can recognize this param and show it as an icon next to the OTP entry.

This is technologically feasible (we'll have to call _generate_uri instead of having PyCA's TOTP implementation do it for us, but that's not a big deal).

However, it might be inadvisable: the otpauth:// scheme isn't standardized (Google provides this spec, which they themselves violate in some slightly annoying ways), and different implementations may balk at an unexpected parameter. FreeOTP's behavior is also underdefined (what image formats are acceptable? how long are images cached for?).

I do think that having an icon makes for a much nicer UX; if we test a few of the most common TOTP applications and can confirm that the parameter doesn't interfere with them, I'd be in favor of adding it.

Disabling it seems too easy: shouldn't PyPI require entering an OTP to complete this action?

This becomes a problem for users who have logged in but no longer have access to their TOTP secret. Some plausible circumstances:

There's some ongoing discussion about requiring a password to disable TOTP in #5825; I'm still okay with the current behavior, but would not be opposed to making the switch over to requiring a password if the PyPI teams 👍 it.

Edit: Typo

brainwane commented 5 years ago

@webknjaz and @woodruffw could the "improve the QR code" discussion move to #5894? And I'd like the "how much confirmation should we require of the user to disable 2FA" discussion to please move to #5825. Thanks.

nurupo commented 5 years ago

This becomes a problem for users who have logged in but no longer have access to their TOTP secret.

What about a user who no longer has access to their TOTP secret and are not logged in?

All 2FA implementations I have seen have a backup method you can use if you lose your TOTP/U2F device. Often times they provide backup codes. Sometimes it's TOTP with SMS as backup, or U2F with TOTP as backup. But I think I have yet to see a 2FA without a backup method.

brainwane commented 5 years ago

@nurupo We are addressing that in #5800 - please head over there! Thanks.

brainwane commented 5 years ago

We've rolled out two methods for 2FA on PyPI: TOTP support and WebAuthn support. The latter is in beta, and #5661 is a meta-issue where we are tracking its rollout and getting the last few items fixed before ending the beta. We don't have enough funding to implement recovery codes; that's in issue #5800 and we're seeking additional funding to pursue that.

Per agreement with other maintainers in a recent meeting I'm therefore closing this issue. Please enjoy 2FA on Warehouse, and file new issues to request new 2FA-related features. Thank you all!