search

pypi / warehouse

The Python Package Index
https://pypi.org
Apache License 2.0
3.59k stars 968 forks source link

2FA/API tokens: staging/testing rollout #5661

Closed brainwane closed 4 years ago

brainwane commented 5 years ago

What's the problem this feature will solve? To finish #996 (see #5567), we need to test MFA with real users on real packages; asking them to spin up dev environments is too hard and won't help multi-maintainer projects reason well about what MFA policies they want to set up.

Describe the solution you'd like My tentative suggestion is:

Additional context

Cc @ewdurbin .

di commented 5 years ago

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.

It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.

This could allow us to do a timeline like:

If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?

Possibly? I think we should be explicit in our "beta" announcement that this might happen, so users aren't surprised.

Are there particular categories of user we need to make sure we get in the beta test?

I think all the groups you listed would be ideal.

people who habitually block a lot of cookies/JS

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency. (cc @woodruffw)

Is test.pypi.org the right place for this?

I think it is. We don't really claim that TestPyPI should be depended on for much.

ewdurbin commented 5 years ago

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.

It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.

This could allow us to do a timeline like:

  • 1 week period on TestPyPI for everyone
  • 2 week beta on PyPI for select users
  • Open to all users

+1 to this approach.

woodruffw commented 5 years ago

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency.

It should be relatively easy to add a fallback here: we can display the otpauth:// URL directly, and the user can copy-and-paste that into their TOTP application or use their QR generator of choice. Right now that URL is only available via the aria-label of the canvas that we render the QR in, but that wouldn't be hard to change at all. (Sidenote: I'm not a frontend developer at all, so I have no idea how to check whether a user has disabled Javascript. Is it just <noscript>?)

This could allow us to do a timeline like:

  • 1 week period on TestPyPI for everyone
  • 2 week beta on PyPI for select users
  • Open to all users

This looks good to me!

brainwane commented 5 years ago

@nlhkabu - does this feel ok to you? If so then I will declare consensus and start planning this, writing sets of "hey come test this" announcement text, etc.

nlhkabu commented 5 years ago

Hi @brainwane - sounds very reasonable to me.

I'd like to run some direct user tests alongside open testing - sessions where I am able to observe users and understand anywhere they get stuck.

I've set up a Google form for registering users for this. @brainwane would you be able to:

  1. review the form
  2. add a link to it in your announcement

To be clear, I don't think that manual user testing should block the launch of 2FA, unless I find any major technical issues that are not raised by open beta testing.

di commented 5 years ago

@nlhkabu One small comment on the form: might want to ask for their PyPI username as well, so we can enable 2FA for them during the private beta.

di commented 5 years ago

5567 is close to being done. I'm going to plan on merging it on or before Friday, May 3rd, which would give us the following timeline:

brainwane commented 5 years ago

OK, draft announcements:

For distutils-sig/discuss.python.org and other longform places:

Subject line: PyPI two-factor auth (2FA) trial May 3-20

Dear PyPI users:

To increase the security of PyPI downloads, we're beginning to introduce two-factor authentication (2FA) as a login security option, and want project maintainers and owners to start testing it.

Starting this Friday, May 3rd, you'll be able to use 2FA on Test PyPI. And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

PyPI currently supports a single 2FA method: generating a code through a Time-based One-time Password (TOTP) application. After you set up 2FA on your PyPI account, then you must provide a TOTP (along with your username and password) to log in. Therefore, to use 2FA on PyPI, you'll need to provision an application (usually a mobile phone app) in order to generate authentication codes; our our testing wiki page gives you suggestions and pointers.

This change only applies to the login step, not package uploads.

More details at our testing wiki page.

During this testing period, if things go awry, there's a chance we will need to wipe tokens from users' accounts, so if you choose to try it, please be forewarned. We suggest you make sure you have a PyPI-verified email address on your user account before trying the feature, to make potential account recovery smoother.

And please let us know if you run into glitches.

We expect to end this testing period on May 20th, then enable the optional 2FA feature for all PyPI users, and move on to working on WebAuthn support.

Thanks to the Open Technology Fund for funding this work. More progress reports at the Packaging Working Group's wiki page.

-the PyPI team

for pypi-announce:

Subject line: PyPI two-factor auth (2FA) trial May 3-20

Dear PyPI users:

To increase the security of PyPI downloads, we're beginning to introduce two-factor authentication (2FA) as a login security option, and want project maintainers and owners to start testing it.

Starting this Friday, May 3rd, you'll be able to use 2FA on Test PyPI. And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

More details at our testing wiki page.

We expect to end this testing period on May 20th, then enable the 2FA feature for all PyPI users, and move on to working on WebAuthn support.

Thanks to the Open Technology Fund for funding this work. More progress reports at our wiki page.

-the PyPI team

brainwane commented 5 years ago

(I need to add a note calling out that this doesn’t affect the upload endpoint currently.)

nlhkabu commented 5 years ago

Looks good to me @brainwane - thank you. You can probably change this sentence to be shorter:

If you set up 2FA on your PyPI account, you must provide a second method of identity verification (other than your username and password) to log in.

mlissner commented 5 years ago

First:

And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

Should this say that it's really more of a user test? It's not just joining a beta, the plan is to sit with these folks for an hour, right? I eagerly clicked the link, then was disappointed since I didn't have the hour.

Second: Is there a plan to require 2FA eventually, or, if there isn't a plan, is it worth threatening that that might happen eventually? I think it makes sense to have 2FA for software repos like PyPi, but I know requiring it would be difficult. Maybe there's a middle ground to start pushing for.

di commented 5 years ago

@mlissner You don't need to be available for that hour to be in the beta, just respond "no". :)

mlissner commented 5 years ago

I'm like 10% sure that changed? Anyway, good!

brainwane commented 5 years ago

When we flipped the switch for this feature on Test PyPI & canon PyPI, we did not initially update the user model so that existing Test PyPI accounts had the feature on. Ernest (I believe) ran some SQL to update all the Test PyPI accounts yesterday and turn on the flag.

Right now, new Test PyPI accounts (as in, created since that time yesterday) do not have the flag set so they don't see 2FA in their account settings.

I'd like @ewdurbin to run the "turn this on for everybody on Test PyPI" SQL again just to get that taken care of, then figure out what to do next.

I think @di told me that turning it on by default for all Test PyPI accounts would also turn it on for all canon PyPI accounts (does this mean we ought to have a better feature flag system in general, or just for this feature?) (and I presume that constraint would also stop us from doing something to fix this like: verifying the email address triggers turning on 2FA support).

ewdurbin commented 5 years ago

Reran the enabling of 2FA on test.pypi.org

brainwane commented 5 years ago

Today's the 20th and it looks like we aren't done with the beta yet, because #5866 is a blocker.

Once we finish that, there are some UI issues that we should fix, but it's ok to roll out 2FA to everyone on pypi.org before fixing them.

And then a note for next time: let's address #5869 (or having "flip bit in user model" in the runbook for the WebAuthN rollout).

brainwane commented 5 years ago

@ewdurbin @dstufft @di do we have any way of knowing how many users on PyPI have turned on 2fa? Is that even a useful statistic?

I naively assume that it would be, and that we would want the number to go up over time, and that we would want a rough estimation in particular of how many users have it turned on who own or maintain at least one package.

(I am ok with the answer being "it is a pain to check that" in which case I will probably ask that we check it, like, nowish and then again in a couple months, as we check how well our staging/rollout/publicity are going. I am of course also ok with the answer being "we cannot know that" or "it is a vulnerability to say this publicly" or what have you.)

ewdurbin commented 5 years ago 
warehouse=> SELECT ur.role_name, 
warehouse->        Count(*) 
warehouse-> FROM   users AS u 
warehouse->        JOIN (SELECT DISTINCT user_id, 
warehouse(>                              role_name 
warehouse(>              FROM   roles) AS ur 
warehouse->          ON u.id = ur.user_id 
warehouse-> WHERE  u.totp_secret IS NOT NULL 
warehouse-> GROUP  BY ur.role_name; 
 role_name  | count 
------------+-------
 Maintainer |    62
 Owner      |   214
(2 rows)
ewdurbin commented 5 years ago

Update as of this time:

warehouse=> SELECT ur.role_name, 
warehouse->        Count(*)    
warehouse-> FROM   users AS u  
warehouse->        JOIN (SELECT DISTINCT user_id, 
warehouse(>                              role_name   
warehouse(>              FROM   roles) AS ur 
warehouse->          ON u.id = ur.user_id  
warehouse-> WHERE  u.totp_secret IS NOT NULL 
warehouse-> GROUP  BY ur.role_name; 
 role_name  | count 
------------+-------
 Maintainer |    88
 Owner      |   326
(2 rows)
brainwane commented 5 years ago

@nlhkabu For the WebAuthn rollout, @ewdurbin suggested that we initially add a badge in the Account Settings marking the WebAuthn 2FA method as "beta". I've thus filed #5976 just now.

ewdurbin commented 5 years ago

Update as of today:

 role_name  | count 
------------+-------
 Maintainer |   105
 Owner      |   446
(2 rows)
brainwane commented 5 years ago

Per our conversation in the Friday meeting we plan to roll out the WebAuthn feature on test.pypi.org and on pypi.org with the "beta" badge (#5976) but without doing a bunch of messy bit-flipping. I've also updated https://wiki.python.org/psf/WarehousePackageMaintainerTesting so it'll be ready to publicize.

Working on copy for blog post(s), mailing list posts, social media, and so on.

brainwane commented 5 years ago

OK, today we're rolling WebAuthn out as beta and asking a few people to test it, and in the afternoon we'll post blog posts and I'll notify distutils-sig et alia. Later this week I'll publicize it to some more communities, and then in maybe 10 days we'll remove the "beta" badge and I'll email pypi-announce and python-announce. Once that is done I figure we can close the issue.

webknjaz commented 5 years ago

@brainwane I've tried that on Test PyPI.

So I have a TOTP set up. I clicked on Add 2FA with security key. It prompted me to enter a Key name which I did (Yubikey Neo).

STR

After that, clicking Provision key does nothing visually. So I've opened DevTools. I can see a successful GET request to https://test.pypi.org/manage/account/webauthn-provision/options with some JSON payload in the response. It looks legit, contains my user data and a challenge. After clicking more times on that button, each of them produces an exception being logged to the JS console. The same happenes on prod PyPI, in incognito mode, with browser extensions disabled.

test-pypi-webauthn-exc
Runtime

Google Chrome Version 69.0.3497.81 (Official Build) (64-bit) running Gentoo Linux

Trace
[8] bind-modal-keys.js:43 Uncaught (in promise) DOMException: A request is already pending.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445
[2] bind-modal-keys.js:43 Uncaught (in promise) DOMException: The operation either timed out or was not allowed. See: https://w3c.github.io/webauthn/#sec-assertion-privacy.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445
brainwane commented 5 years ago

Thanks for the bug report @webknjaz - I've turned that into #6050 so we can address it there.

brainwane commented 5 years ago

We've had a few issues (such as #6106) due to a misconfiguration of test.pypi.org, and we're getting more user testing now thanks to outreach by @nlhkabu (such as this email to the Education SIG which I forwarded to Software Carpentry's discussion list). I'm going to wait till we have a few more data points from Nicole's testing, so we can fix issues that crop up. Then I'll call the beta finished and we should make further public announcements.

nlhkabu commented 5 years ago

Thanks @brainwane. FYI, I have user testing scheduled most days in the coming week. The final session is scheduled for Tuesday 9th.

brainwane commented 5 years ago

@nlhkabu ran user tests, summed up her findings in #6173, and we've now addressed those issues, yay! But we still have a few WebAuthn issues left in the milestone before I'll call the beta finished.

Once we approve #6084 we'll have one more auth feature -- API tokens -- to test via a beta period on test.pypi.org and pypi.org, so I'll keep this issue open to track that, then close it.

brainwane commented 5 years ago

@ewdurbin May I ask:

ewdurbin commented 5 years ago

@brainwane I'll query for those numbers before lunch, unfortunately we don't have this in Datadog at the moment, we'd need to implement a scheduled task to populate them... which perhaps is worthy of doing.

ewdurbin commented 5 years ago
  • how many users have turned on 2FA at all?
warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count 
-------
  1688
(1 row)
  • how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP

warehouse=> select count(*) from users where totp_secret is not null;
 count 
-------
  1632
(1 row)

WebAuthn

warehouse=> select count(distinct user_id) from user_security_keys ;
 count 
-------
   132
(1 row)
  • what proportion & number of logins over the past week were protected by 2FA?

This we may be able to get from Datadog...

  • how many API tokens currently exist?
warehouse=> select count(*) from macaroons ;
 count 
-------
    21
(1 row)
ewdurbin commented 5 years ago

Past Week:

Percentage of Logins using Two-Factor Authentication: 7.71%

Screen Shot 2019-07-26 at 1 13 35 PM
brainwane commented 5 years ago

Edited Sept. 8th to start using the beta blockers milestone to track the stuff below.

Issues we need to resolve before ending WebAuthn beta:

Issue(s) we need to resolve before ending API token beta:

After the beta's done:

brainwane commented 5 years ago

Percentage of Logins (to pypi.org via browser, I think) using Two-Factor Authentication:

May: 2.25% June: 3.08% July 1-21: 4.62% July 21-25: 9.72% (July 25th was our post to Discourse about API tokens) July 25-28: 15.45%

And, for logins in the past 2 days, it's 23.1%. That is mostly a weekend, so it'll be interesting to see whether the trend changes on weekdays...

takluyver commented 5 years ago

Is there any plan for an API to create upload tokens? E.g. I'd like to have a command-line tool prompt me once for my password & 2FA code, then obtain and store a project-scoped token to use for uploads.

ewdurbin commented 5 years ago

We have updated the token username and prefix in #6342.

username: @token => __token__ password/token: pypi:<base64 token body> => pypi-<base64 token body>

These changes should alleviate the need for escaping heroics.

The previous format will continue to work for now, but users will be notified to update their configurations to match the new syntax before the beta period is over.

brainwane commented 5 years ago

Contractors on the OTF-funded work need to stop/deprioritize work on the security features in order to ensure we complete the accessibility and internationalization work by the end of the month. Therefore, even though some security features are still in beta, I'm closing the milestone.

brainwane commented 5 years ago

I'm writing a discuss.python.org/distutils-sig post now to update our community on the end of the OTF-funded work. I'd like for us to have a few more beta blockers closed before we send out a pypi-announce email, to reduce how much time we and other volunteers spend on support issues; it's ok with me if that means the email doesn't go out for a few more weeks.

ewdurbin commented 5 years ago

Update for today:

  • how many users have turned on 2FA at all?
warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count 
-------
  3544
(1 row)
  • how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP

warehouse=> select count(*) from users where totp_secret is not null;
 count 
-------
  3398
(1 row)

WebAuthn

warehouse=> select count(distinct user_id) from user_security_keys ;
 count 
-------
   336
(1 row)
  • how many API tokens currently exist?
warehouse=> select count(*) from macaroons ;
 count 
-------
  1127
(1 row)
brainwane commented 4 years ago

Almost ready for API tokens to leave beta! Just waiting to land https://github.com/pypa/packaging.python.org/pull/687 .

@ewdurbin may I ask for an updated count, on how many users have turned on 2FA at all, and how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

brainwane commented 4 years ago

My current draft of an announcement email:

Subject: Start using 2FA and API tokens on PyPI

Dear PyPI users:

To increase the security of PyPI downloads, we have added two-factor authentication (2FA) as a login security option, and API tokens for uploading packages.

If you maintain or own a project on the Python Package Index pypi.org , you should start using these features. Click "help" on PyPI for instructions.

Details and plans for the future:

2FA: PyPI's implementation of the WebAuthn standard means you can use any 2FA device that meets the FIDO standard. 2FA only affects logging in via a web browser, and not (yet) package uploads.

API tokens: use these (instead of username and password) to authenticate when uploading packages to PyPI. You can make tokens that work for all your uploads. You can also make tokens whose scope is limited to one specific package. That way, if a token is compromised, you can just revoke and recreate that token, instead of having to change your password in lots of automated processes.

For more details and instructions, click "help" on PyPI: https://pypi.org/help/ . (These features are also available on Test PyPI.)

In the future, PyPI will set and enforce a policy requiring users with two-factor authentication enabled to use API tokens to upload (rather than just their password, without a second factor). We do not yet know when we will make this policy change.

Thanks to the Open Technology Fund for funding this work. More work is in progress on pip and PyPI -- see https://wiki.python.org/psf/PackagingWG .

-Sumana Harihareswara on behalf of the PyPI team

ewdurbin commented 4 years ago

Update as of this moment:

  • how many users have turned on 2FA at all?
warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count 
-------
  5362
(1 row)
  • how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP

warehouse=> select count(*) from users where totp_secret is not null;
 count 
-------
  5090
(1 row)

WebAuthn

warehouse=> select count(distinct user_id) from user_security_keys ;
 count 
-------
   571
(1 row)
  • how many API tokens currently exist?
warehouse=> select count(*) from macaroons ;
 count 
-------
  3232
(1 row)
brainwane commented 4 years ago

I've made the launch announcement on pypi-announce and thus I now declare this issue closed. Thank you, everybody.