Open jenshenny opened 2 years ago
indirect
commented
2 years ago npm has much higher absolute numbers than we do, but starting somewhere in the (say) top 5 or 10 percent of gems seems reasonable to me. I don't actually know what percentage of all gems are personal projects, or what the distribution of dependents or downloads looks like. Is that something we can gather and visualize here before making a decision?
hsbt
commented
2 years ago We need to discuss the support cost for recovery action by people who lost the device.
jenshenny
commented
2 years ago @indirect Yes definitely, that would be super helpful in making some decisions.
Some key general metrics... Total number of gems ~185,000, Total download count ~103 billion.
I attempted to visualize the download distribution months ago by making a histogram of download count per gem but it was very difficult in providing much insight. A better way was to plot the top x gems and determine the proportion of the total download count it occupies. The download count of the nth gem (to determine a potential download threshold) was also gathered.
number of gems (most - least downloaded) | Percent of Total Downloads | total downloads of nth gem
-- | -- | --
1 | 1.06830509 | 1108638775
5 | 3.729372391 | 679957085
10 | 6.772211241 | 598432948
25 | 14.59134825 | 482443362
50 | 23.69050621 | 304843640
75 | 30.28665061 | 244406076
100 | 35.54329822 | 194131166
250 | 52.65508468 | 7736359
500 | 66.82922319 | 49407255
750 | 76.24105138 | 30444937
1000 | 82.01376667 | 19045789
2000 | 91.39766826 | 4725007
3000 | 94.37307488 | 1963052
4000 | 95.71925393 | 1008652
5000 | 96.47650842 | 607692
7500 | 97.4092375 | 249010
10000 | 97.85644235 | 140501
25000 | 98.81773161 | 35843
50000 | 99.35920433 | 14663
75000 | 99.6264661 | 8376
100000 | 99.78665053 | 5216
125000 | 99.88867902 | 3469
150000 | 99.95875191 | 2428
180000 | 99.99982382 | 103
Based on the results above, the percentage really plateaus with the top 50 000 gems (99%). However, 95+% of total downloads is captured with the top 5000 gems so that might be more impactful. Wondering if I'm missing any information that would provide more insight.
I haven't taken a look at analyzing dependents closely but am planning to shortly. Just wanted to raise from the original RFC, there has been some concerns about this approach as to be truly secure, transitive dependencies from all versions (not just the latest) of the top x gems would have MFA required which could slowly spiral out of control in regards to computation.
jenshenny
commented
2 years ago @hbst right, that's a good point. I assume the current process is to email support and manually reset their MFA which could be tedious if there are many requests.
indirect
commented
2 years ago I think I'm fine with enabling MFA only on the current and future versions, but even if we wanted to enable it on past versions it would be a one-time computation on a fixed size graph.
As for how many gems to enable MFA on, I think even 600k downloads all time is probably too high—how many installs are those gems getting on a weekly basis? Based on these numbers, I would suggest something like aiming for the top 2000 gems, phased in (say) 500 per month, or something like that.
indirect
commented
2 years ago As a random example to illustrate my point, here is a gem with 610,487 all time downloads: https://rubygems.org/gems/oai. The most recent version was released April 29. In the last 8 weeks, that version has gotten 671 downloads. I don't think it should require MFA.
sonalkr132
commented
2 years ago Downloads count can definetly me misleading and we don't track downloads over a period (just total downloads).
transitive dependencies from all versions (not just the latest) of the top x gems would have MFA required which could slowly spiral out of control in regards to computation.
I think I'm fine with enabling MFA only on the current and future versions
I created a Gemfile with top 100 gems and used bundle install (assuming this would only pick latest versions, as mention in my referenced comment this method has nuances). I got total of 441 gems, out of 316 are just aws-... For remaining 125 gems, only 11 owners don't have mfa enabled (where gem had a release in last three years). This should have a significant overlap with top 100 owners. An offline process to mark these 441 gems to require mfa seems like a reasonable step for next phase. I would also ensure that mfa requirement list is append only. ie we won't remove gems from list once added even when above condition is no longer met.
jchestershopify
commented
2 years ago In terms of supporting MFA resets, I know this is a problem for our peers in other ecosystems. There's an idea circulating to ask OpenSSF to fund a position (1 or more, we will have to see) for support techs to be shared between ecosystems. I think it would be worth participating in such a scheme -- and saying so when it comes up for OpenSSF board consideration.
jenshenny
commented
2 years ago It is true that total downloads doesn't provide any information about the gem's activity in a period of time which makes it not the best metric to measure.
I looked at the change in download count between June 16-23 and June 9-16 and averaged those results from those 2 dates. The avg weekly download count of the top 2000 gems hovers around 30k while the top 5000 hits below 3k per week.
I also checked the runtime dependencies of the top 100 gems and found that out of ~400 total gems, 45 gems have less than 5 million total downloads (most are around 2 million to 4 million). I checked their weekly downloads and they all have weekly downloads over 70k.
We could run an offline process to determine transitive dependencies and require the ~400 gems. Alternatively, I think we could require the top n gems, start tracking the downloads over the most recent week, and for gems that surpass >70-100k downloads over the week, we would also require them to enable MFA.
I would also ensure that mfa requirement list is append only. ie we won't remove gems from list once added even when above condition is no longer met.
+1, a follow up to that is what would it look like when gems are added to the list? I don't think it would be great to enforce MFA immediately, maybe give a time period of a month where we'll send an email notification, warnings would show up before MFA is required.
rdp
commented
1 year ago I'm confused why specific_install gem seems to be requiring MFA with 4M downloads? Maybe because a different gem associated with my username has passed the threshold? Maybe could mention that in the blog? Thanks :) Might want to mention the version of rubygems required for pushing with it in the docs? Thankfully it seems possible to just run it from the command line since I go through phones like water :) https://serverfault.com/questions/519956/is-there-a-command-line-two-factor-authentication-verification-code-generator/519961#519961
simi
commented
1 year ago I'm confused why specific_install gem seems to be requiring MFA with 4M downloads? Maybe because a different gem associated with my username has passed the threshold? Maybe could mention that in the blog? Thanks :) Might want to mention the version of rubygems required for pushing with it in the docs? Thankfully it seems possible to just run it from the command line since I go through phones like water :) https://serverfault.com/questions/519956/is-there-a-command-line-two-factor-authentication-verification-code-generator/519961#519961
In your case it seems os gem is the reason why your account has MFA enforced right now. It is mentioned in the blog at https://blog.rubygems.org/2022/08/15/requiring-mfa-on-popular-gems.html.
Today (August 15th, 2022), we will begin to enforce MFA on owners of gems with over 180 million total downloads.
You can find details on CLI usage at https://guides.rubygems.org/using-mfa-in-command-line/.
The RFC to require MFA on accounts that own a gem with over 180 million downloads has been accepted and currently is being rolled out :tada: In the last section of the RFC, it stated that
This issue is being opened to create a discussion on how we should implement this phase in the current MFA rollout! The ideas formed here will guide what will be drafted in a formal RFC.
Topics to discuss
Other package ecosystems
Npm has rolled out the following policy: