Proposed Initiative: "Maintainer Experience" of OpenSSF

[webchick]

(I recently attended Open Source Summit in Vancouver, and this issue is based on a convo with Anne Bertucio from Google, who directed me to post an issue here. Apologies in advance if there is a better place / better way to go about sharing this feedback!)

Context: I'm a long-time core maintainer and general cat herder of the Drupal open source project, a member of their security team, and a general security enthusiast since the alt.2600 days. I sort of accidentally learned about OpenSSF's existence last year (and also accidentally learned we are deemed a "critical project"), and since then have spent a few weekends digging around the website and reading a whole bunch. I've attended a couple of town halls, and attended the morning of OpenSSF Day a couple of weeks back. I would rate my knowledge about open source maintainership 9/10, my OpenSSF knowledge maybe 3/10.

So with that in mind, gulp, here goes. :)

---

Problem Statement

Right now, OpenSSF's activities are presented in a way that is congruent with other Linux Foundation projects: there's a strong focus on the how the OpenSSF is structured, how the governance works, how it is run day to day, what are the recent achievements, and so on. All of that makes total sense, through the lens of being a member-driven Linux Foundation project, and wanting to demonstrate your value to prospective members.

However, by and large, open source maintainers:

• Have very little time for "extracurricular activities" beyond their maintainer responsibilities, since they are often balancing these with a job and a family
• Therefore, tend to "silo" into their own projects, their own community, and their own problems, by necessity
• Therefore, be broadly unaware of OpenSSF's existence — if they go to events, and have $1000 to spend on a ticket, they will tend to attend something like DrupalCon, not something like Open Source Summit
• Once learning of OpenSSF's existence, they will have very little time to dig through information (including terms/acronyms they may not already be familiar with, such as "SBOMs") to answer the question "What are the problems you solve for my project and how do I partner with you?"
• When they attempt to answer this question based on the available information, they'll see a lot of evidence of how OpenSSF helps enterprises, governments, etc. and their customers, but not a lot of information of how it helps them as an everyday maintainer. (They may in fact leave with the opposite impression, that the OpenSSF exists to make their job harder, in order to please enterprises / customers :\ — see the blow-up when YubiKeys were distributed to maintainers of all the top Python projects, for example.)

Objectives

Taken together, it seems like in order for OpenSSF to broadly win the "hearts and minds" of maintainers, there needs to be focus on (at least) the following things:

• We need to meet maintainers where they are in order to raise awareness of OpenSSF.
• We need to ensure they achieve this understanding in as little time as possible.
• We need to present the activities of OpenSSF in a pain point/solution-focused way, to help make our benefits tangible and demonstrate empathy with their needs.
• We need to ensure there's proactive, ongoing engagement with maintainers and their projects; "once and done" is too easy to get buried under the day to day demands.

(This effort could dovetail nicely with the work going on in #165)

Proposed Solutions

There are many ways to go about this, and I'm sure you would know better than me what makes sense :), but here are some initial thoughts:

"Maintainer Experience" Working Group

Create an OpenSSF Working Group specifically around maintainers and their needs (similar to the End Users group, but for the producers of open source).

This could also manifest as something similar to the TAC but a committee made up of maintainer representatives. (Just watch the time commitment.)

Dedicated "Landing Page" Specifically Aimed At Maintainers

For example, some sort of "call to action" on the OpenSSF website "Maintainer? Click here!" and/or maybe repurpose https://github.com/openssf (which currently appears empty) for this purpose?

The contents of this page would be geared toward maintainers and their pain points, rather than oriented around how OpenSSF thinks of them. (I can assure you that maintainers don't care whether something is a working group or a project ;)).

For example:

• Want to learn best practices in secure coding? Take our free training.
• Interested your project being automatically scanned for vulnerabilities? Learn more about Alpha-Omega
• Want to ensure your end users are getting the software distributed to them that they're counting on? Check out Sigstore.
• ... (etc.) ...
• Want to talk to us to find out more? Join a dedicated Slack channel

Oh, speaking of which...

Dedicated Slack Channel (+Mailing List?) Aimed At Maintainer Relations

Upon joining OpenSSF Slack I was blown away by the calibre of people there. :O However, if I was entering that space as a maintainer, I would quickly reach the conclusion that this is not a space for me, because the vast majority of conversation in there is about promotion/enablement of OpenSSF itself, the Working Groups' day to day toils, etc.

So some sort of dedicated channel(s) where maintainers can come in, introduce themselves, ask questions related to OpenSSF's activities, get answers from someone, meet and network with other security-conscious open source maintainers. Maybe also a corresponding "Maintainer Office Hours" meeting folks could join, and periodic "What is openSSF for maintainers" onboarding sessions (could re-use these at events).

Forge Partnerships With Open Source Security Teams

Open source security projects' teams are overwhelmed and exhausted. Having partnership from an organization like OpenSSF would be a huge shot in the arm, and an excellent, pragmatic way to demonstrate OpenSSF's value in a way maintainers can fully understand.

What could this look like? Maybe a set of "tiger teams" grouped by programming language who prioritize projects with massive ecosystems, helping them to implement things like sigstore, teach best practices like CVEs, etc.

Go Where Maintainers Are

In each of the critical projects (once again, probably starting within those with massive ecosystems, such as Python), figure out where those maintainers congregate in "real life" (PyCon), and go there. Even better: co-present with the security maintainers from the previous step; now you have automatic credibility within that community, and a reason for people to show up to your talk.

Assume Zero Knowledge In Community Events

In Town Halls, OpenSSF Day, etc. always assume someone in the audience is hearing about OpenSSF for the first time (if you're doing lots of outreach, they probably are!). The first time you use terms like "SBOM" define for people what that is (and/or have a handy glossary page that's distributed ahead of time). When someone gives a status update on Sigstore, give 30 seconds of context as to what that is and why it's important.

Without this, the risk once again is giving off the strong vibe that OpenSSF is for "insiders" and not for maintainers themselves.

Thoughts?

Haha, sorry for the novel, apparently there was a lot to say. ;) Does the problem statement ring true to you? Is there already an initiative underway in OpenSSF that maintainers interested in smoothing these sorts of partnerships could join and participate in? Is this far out of what OpenSSF sees as their mission and better for some other group to tackle?

Very curious of your thoughts!

[lehors]

Thank you for taking the time to not only report the problems you are seeing but also make concrete suggestions on how we might go at addressing them. This is very valuable!

We do have one WG that is maintainers oriented: Securing Software Repositories https://github.com/ossf/wg-securing-software-repos

Have you looked into this? I'm curious to know whether you didn't run into it (which wouldn't surprise me, there is a lot going on and it's not easy to know what's out there) or if you know about it but didn't think it fits the bill. If so, why?

Thanks again!

[david-a-wheeler]

@lehors - Securing Software Repositories is definitely worth looking at. However, that WG focuses on "maintainers of software repositories, software registries, and tools", e.g., PyPI/pip, npm, RubyGems/bundler, system software registries / {apt-get,dnf}. So that WG does not directly apply to the maintainers of most OSS projects. Of course, most widely-used OSS projects end up in some repo, so a typically OSS maintainer would want them to work well, but that's not the same thing.

The best practices WG generates a number of guides & materials specifically for maintainers, and I think that's the closest match at this moment... but it often focuses on work we're doing, not "here are finished products you can use".

There are a lot of interesting proposals here! A "landing page" for maintainers, in particular, sounds like a great idea to me. We could point to our guides, the education materials, sigstore, etc.

[SecurityCRob]

I think that the “maintainer/developer” persona is one of the core foci of our stalwart band of security-friends. I love the idea of a landing page, and materials dedicated to helping kickstart or assist a developer in their personal security journey. I agree with David, the BEST WG feels like a slightly better fit, and we even can augment some of our targets for the EDU.SIG to address getting materials composed/collated/presented for this group.

Cheers,

CRob Director of Security Communications Intel Product Assurance and Security

From: David A. Wheeler @.> Sent: Tuesday, May 30, 2023 9:47 AM To: ossf/tac @.> Cc: Subscribed @.***> Subject: Re: [ossf/tac] Proposed Initiative: "Maintainer Experience" of OpenSSF (Issue #169)

@lehorshttps://github.com/lehors - Securing Software Repositorieshttps://github.com/ossf/wg-securing-software-repos is definitely worth looking at. However, that WG focuses on "maintainers of software repositories, software registries, and tools", e.g., PyPI/pip, npm, RubyGems/bundler, system software registries / {apt-get,dnf}. So that WG does not on the maintainers of most OSS projects. Of course, most widely-used OSS projects end up in some repo, so a typically OSS maintainer would want them to work well, but that's not the same thing.

The best practices WGhttps://github.com/ossf/wg-best-practices-os-developers generates a number of guides & materials specifically for maintainers, and I think that's the closest match at this moment... but it often focuses on work we're doing, not "here are finished products you can use".

There are a lot of interesting proposals here! A "landing page" for maintainers, in particular, sounds like a great idea to me. We could point to our guides, the education materials, sigstore, etc.

— Reply to this email directly, view it on GitHubhttps://github.com/ossf/tac/issues/169#issuecomment-1568467090, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQRFDLD37V47WIYKFUEZMGLXIX26FANCNFSM6AAAAAAYS7XDVU. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

[webchick]

Right, echoing some of the other comments here:

• My understanding of https://github.com/ossf/wg-securing-software-repos is they would be engaging more "upstream" from a typical open source maintainer/developer. In the Drupal world, it seems like the types of folks they would engage would be our Drupal.org infrastructure team (~5 people) who manage our GitLab instance and maintain various packaging scripts. Not the 30K module maintainers writing Drupal code who are the ones actually creating the security problems in the first place. ;)

• https://github.com/ossf/wg-best-practices-os-developers indeed seems like a closer fit, but the "problem" (and again, this is not a problem per se, but a lack of alignment in needs for this specific audience) is the README there is very focused on "how the working group gets things done," what's the broader, holistic approach they're taking to achieving their aims, and their Slack channel (as is common with all of the WG Slack channels) consists mostly of a lot of "tactical" chatter amongst the WG members about rescheduling meetings or setting GitHub permissions, vs. a bunch of conversations with open source maintainers themselves, sharing best practices/advice to them directly.

Does this distinction make sense?

[webchick]

Also, thank you so much for taking the time / effort to read and consider this, and also for your patience as I muddle my way through an unfamiliar process. :)

[di]

My understanding of https://github.com/ossf/wg-securing-software-repos is they would be engaging more "upstream" from a typical open source maintainer/developer. In the Drupal world, it seems like the types of folks they would engage would be our Drupal.org infrastructure team (~5 people) who manage our GitLab instance and maintain various packaging scripts. Not the 30K module maintainers writing Drupal code who are the ones actually creating the security problems in the first place. ;)

As chair of this WG, +1, absolutely right!

[SecurityCRob]

We are VERY happy to have you participating to make our efforts more accessible for everyone! Thank YOU, Angie!!

[lehors]

Ok, I can see how the Best Practices WG might be a better fit. This WG has primarily been oriented towards developing material like guides though. I wonder what's better: focus that WG more towards the maintainers needs as described by @webchick or create a new WG dedicated to just that.

But for that matter I agree with the idea of adding a dedicated landing page idea and in fact it will help people find their way to whichever WG we decide is where they should go. :-)

[annabellegoth2boss]

I think the persona (to issue #165) with the problem @webchick describes is a bit different than the Best Practices WG. As I understand it, Best Practices WG is about content development for the 100k+ developers writing code that's open source, and not knowing they should do something like set up GitHub Branch Protection from the get-go.

This seems to be about what and how the OpenSSF helps an "OSS Core Maintainer." For example, sigstore was set up for Kubernetes because lead participants of the relevant k8s SIG were interested and kicked it off. That kind of difference makes me think this is a new effort (which ties into similar "maintainer/contributor experience" problems the @ossf/governance-committee has been discussing, so I think we've got some kismet here...)

[TheFoxAtWork]

As additional endorsement on the need for this, when the OpenSSF Maintainer Workshop was held (invite only, small audience and also whose intent was to learn and capture these sentiments), the need articulated here, the observations and experience summarized are all key points that arose in those discussions.

Unfortunately (like many things) the volunteers who worked to execute the event where very limited in their time and couldn't commit to provide a comprehensive analysis and write up from all the notes we captured (even though we really wanted to) - but we did capture notes that would potentially be beneficial as an initial gauge on interactions and engagement with maintainers. @SecurityCRob and @bobcallaway were both excellent Moderators in those discussions. Those notes still exist but will need scrubbed a bit as the event was under Chatham House Rules. Once that is done it could be provided publicly.

If you were part of that effort, please let me know if anything i stated is misrepresented as my participation was time-limited and i may not have the follow-on actions correct.

[DianaOlympos]

As a maintainer of open source that has been looking at the OpenSSF for quite some time now, I will support what @webchick said.

Most of the "best practices" presented by the OpenSSF are, quite directly, a no-go. At this point, I have written off all that comes out of the OpenSSF and the wider "securing supply chain" ecosystem as "the classic infosec that never matters for us or tries to help us" and I am not alone, most of the maintainers I know have made this analysis too.

I want to be clear. I am not saying that because I want to take down the OpenSSF. I say that to give you real feedback from the ground that what @webchick is doing is being really nice to tell you that, in its current shape and operation, the OpenSSF did not manage to talk to "weekend warriors" maintainers and did not manage to set roots in these communities.

Her recommendations and the work on personas are definitely a path to getting better results there. Note in particular that k8s is definitely not representative of the "weekend warrior" persona as a group of maintainers and contributors 😄 They do a great job, just really different persona.

[SecurityCRob]

I'll loop Katherine Druckman in today during the Governance committee meeting. She is helping lead our new DevRel committee that seems to be an excellent place to address @webchick 's observations and concerns. BEST WG will be glad to collaborate on assisting to advocate for the developer/maintainer perspective, and we'll seek to get other groups to participate too.

[lucasgonze]

I also come to OpenSSF from a maintainer perspective. I'd be happy to participate in follow-up work.

[david-a-wheeler]

All: Here's cut at a "Developer landing page": https://best.openssf.org/developers

Suggestions welcome.

We could link to it from the top-level openssf.org page if we think it's useful.

This page does not resolve this issue, but I think it's a step on the path.

[david-a-wheeler]

FYI, there's a best practices WG issue to vote to post the "developer landing page" on the main OpenSSF website: https://github.com/ossf/wg-best-practices-os-developers/issues/195 That doesn't resolve all the issues noted here, but it's a step in that direction.

[joshuagl]

This page does not resolve this issue, but I think it's a step on the path.

Agreed, this is a great start. Could we also link to s2c2f? Or perhaps we want a separate "for organisations" page that would link to s2c2f? 🤔

[SecurityCRob]

related to https://github.com/ossf/tac/issues/84

[evverx]

At this point, I have written off all that comes out of the OpenSSF and the wider "securing supply chain" ecosystem as "the classic infosec that never matters for us or tries to help us" and I am not alone, most of the maintainers I know have made this analysis too

To be fair the fuzzing branch is actually useful. They actually talk to actual maintainers, try to cover even unusual use cases (if it's doable) and so on. Unfortunately snapshot fuzzing (or any other custom stuff tailored to specific projects) is unlikely to ever be supported there but it isn't the end of the world if people fuzzing projects have their own clusters. That being said that branch existed long before OpenSSF was created so it's probably safe to say that it isn't representative of OpenSSF in general.

a bit different than the Best Practices WG. As I understand it, Best Practices WG is about content development for the 100k+ developers writing code that's open source

Agreed. That WG group appears to operate on the assumption that everyone is uneducated and has no clue what they are doing and that sort of attitude isn't exactly helpful if the idea is to win the "hearts and minds" of maintainers.

I'm not sure the proposed solutions can work as long as the "consumer" persona is much more important to OpenSSF but it's good to know that it's at least trying to move in the right direction.

make our efforts more accessible for everyone

It would be great if it was possible to download things like https://openssf.org/soss-vision-brief/ without filling out those forms.

[lucasgonze]

I think the WGs that work on Scorecard and the Badge don't entirely agree on their purpose and meaning, and I personally don't agree at all with the consensus, yet I use these almost every day. What isn't useful is the idea of an objective quantity of security, because the accuracy is too low to make a big difference.

What is very useful is guiding my roadmap as a security TPM for FOSS projects. I treat the criteria as checklists. By running the metrics I figure out what I do and don't need to work on. For example, this month I am cleaning up CodeQL alerts as a result of working through https://bestpractices.coreinfrastructure.org/en/projects/6358#analysis and getting a failing score on the static_analysis_fixed metric.

What I would want from the WGs is to reframe their outputs as a guide for security-related workflows. For example there could be a Kanban template with the first swim lane prefilled with cards corresponding to metrics. I have something similar for OpenChain, which has a comparable role for getting an OSPO organized.

[evverx]

What I would want from the WGs is to reframe their outputs as a guide for security-related workflows

Agreed. I think the problem is that they were never designed to be integrated into maintainers' workflows. Up until recently scorecard just dumped json and ignored maintainers' feedback completely. The human-friendly UI was added eventually but it took a weird amount of time and at least one critical project planning to throw scorecard out. The scorecard action is useless in terms of actually preventing issues and so on.

(In theory this discussion can be moved to the corresponding bug trackers but I don't think it's going to be addressed. They no longer auto-close issues though so at least it can be kept there)

[SecurityCRob]

Agreed. That WG group appears to operate on the assumption that everyone is uneducated and has no clue what they are doing and that sort of attitude isn't exactly helpful if the idea is to win the "hearts and minds" of maintainers.

That is not accurate. The group works under the assumption that there are multiple levels of "learners" and consumers of OSS. The three main developers we think about are beginners/people that are new to the trade, people that are changing careers into development, and existing experienced developers. The current available class focuses on beginning software engineers (although more experienced developers could benefit from that content as well as they may have not been exposed to it before). There are materials being developed on intermediate and advanced development topics that will be available in the future. Patches, RFEs, and comments are always welcome by the group for new projects to help the community.

The other materials the group has produced (Concise Guides, SCM Guide, C/C++ Guide) are usable by any developer, operator, or consumer of open source.

The Scorecard project is currently working on making integrating the checks in a more user-friendly manner for devs and consumers amongst a host of other bug fixes and enhancements. You can check out their current backlog here: https://github.com/ossf/scorecard/issues

[SecurityCRob]

What is very useful is guiding my roadmap as a security TPM for FOSS projects. I treat the criteria as checklists. By running the metrics I figure out what I do and don't need to work on. For example, this month I am cleaning up CodeQL alerts as a result of working through https://bestpractices.coreinfrastructure.org/en/projects/6358#analysis and getting a failing score on the static_analysis_fixed metric.

What I would want from the WGs is to reframe their outputs as a guide for security-related workflows. For example there could be a Kanban template with the first swim lane prefilled with cards corresponding to metrics. I have something similar for OpenChain, which has a comparable role for getting an OSPO organized.

Thank you for the feedback. I'll put this on the docket for the group to discuss in Jan2024. We certainly can spin up a group to work on crafting a checklist/guide to deliver something as you suggest.

In the meantime, the group would welcome your feedback in a future WG meeting or perhaps an issue describing your desired artifact that we could collaborate on together: https://github.com/ossf/wg-best-practices-os-developers/issues

[evverx]

That is not accurate. The group works under the assumption that there are multiple levels of "learners" and consumers of OSS.

As far as I can remember my comment was inspired by a thread where it was discussed that most developers aren't taught how to develop secure software and have no idea what the principle of least privilege is or something like that. I agree I overgeneralized.

Either way I replied to https://github.com/ossf/tac/issues/169#issuecomment-1570166845 and my point is that that WG isn't the place where core maintainers can describe their pain points and get actual things done. For example it's unlikely that that WG can arrange another cluster for people fuzzing a gazillion open source projects.

Patches, RFEs, and comments are always welcome by the group for new projects to help the community.

I don't doubt that patches are always welcome but I'm afraid I don't have time for that. I contribute to projects like OSS-Fuzz and Fuzz-Introspector because they actually help me to address actual issues I have. I contributed to scorecard too but it was because it produced a lot of false positives and it was easier to fix that stuff than to wait for the official fixes.

You can check out their current backlog here: https://github.com/ossf/scorecard/issues

I'm aware where their repository is but thanks.

[lucasgonze]

the group would welcome your feedback in a future WG meeting or perhaps an issue

See https://github.com/ossf/wg-best-practices-os-developers/issues/344

@SecurityCRob It would be valuable to put this on the agenda of a specific meeting. Do you have thoughts on which WG that would be?

Would the Best Practices for OSS Developers WG be the right fit?

[DianaOlympos]

That is not accurate. The group works under the assumption that there are multiple levels of "learners" and consumers of OSS. The three main developers we think about are beginners/people that are new to the trade, people that are changing careers into development, and existing experienced developers. The current available class focuses on beginning software engineers (although more experienced developers could benefit from that content as well as they may have not been exposed to it before). There are materials being developed on intermediate and advanced development topics that will be available in the future. Patches, RFEs, and comments are always welcome by the group for new projects to help the community.

That is great, and I laud this. More and better educational materials are always great. It will really help for new projects made by newcomers that are already safety-critical and... Wait. These do not exist. Right. I mean, they probably will at some point, and there are probably some. I also deeply wonder what kind of educational material for "advanced" maintainers you expect would make a difference. Is there really a well of knowledge that experienced maintainers do not already have access to and need about "how to make safe code?" with things we have not been told hundreds or thousands of times for the past 2 to 3 decades?

Like, I get the point of writing these Best Practices; it is really helpful to show policy makers and maintainers to enforce compliance. You need the list to score and enforce the rules. I get which crowd the OpenSSF is talking to here. But that is not maintainers. And so, I will support once more a "Maintainer Experience" group, at least in order to publish data and research on it. We have scant work reporting on Maintainer Experience. The Road and Bridges report, a few research papers, and the Tidelift survey come to mind.

I think this would be something where the OpenSSF could have an important impact in making visible the realities of the sharp end and how it affects security instead of only reacting and trying to make sense of the millions of packages and their security.

[evverx]

Is there really a well of knowledge that experienced maintainers do not already have access to

There are better places to learn things but they can't be easily linked to maintainers/projects to evaluate/certify them. I think some sort of rationale can be found at https://openssf.org/oss-security-mobilization-plan/ where Stream 1 is supposed to "Deliver Baseline Secure Software Development Education and Certification to All". scorecard should start downgrading projects with no OpenSSF "credentials" soon as far as I can remember :-)

And so, I will support once more a "Maintainer Experience" group, at least in order to publish data and research on it

Seconded.

[david-a-wheeler]

The OpenSSF website tab "Technical Initiatives / For Developers" specifically links to the "For Software Developers" landing page, which tries to provide a short list of materials ready for use directly by software developers.

I'm sure it can be improved, but the intent is for that to help (pull requests welcome). Hopefully that makes a dent in the need to improve maintainer experience.

Is there really a well of knowledge that experienced maintainers do not already have access to and need about "how to make safe code?" with things we have not been told hundreds or thousands of times for the past 2 to 3 decades?

Let me talk a little about education on developing secure software. My experience, and yours may differ, is that many maintainers of important software have never heard about how to develop secure software in enough detail to do anything about it. Most universities & bootcamps don't teach how to develop secure software, and "learning from your peers" only works when many peers already know it. If you're in the security world you're well aware of the issues, but that doesn't mean most software developers know. Here's some evidence for this belief:

1. 53% of software developers report that their organizations don’t ensure training on securing coding [Poneman 2020]
2. No top 40 US “coding” or top 5 non-US CS school required secure coding in 2019 [Forrester 2019]
3. Of U.S. News's top 24 CS 2022 schools, only 1 requires security for undergraduates.
4. The third most popular answer for how to improve OSS security was providing more training to the OSS community. Source: the 2022 v2.0 survey “Addressing Cybersecurity Challenges in Open Source Software” by Stephen Hendrick (VP Research, The Linux Foundation) & Martin McKeay (Senior Editorial Research Manager, Snyk), question q0050mrv. The only higher-ranked items were “define best practices for secure software development” and “provide tools for analyzing and remediating vulnerabilities in the top 500 open source components” - which clearly don’t conflict with training.
5. One article pointedly noted, “universities don’t train computer science students in security”.
6. One survey claimed otherwise, but it is misleading. The State of Developer-Driven Security Survey, Secure Code Warrior, 2022, found that 89% of developers reported they’ve received sufficient training in secure coding skills. However, what this survey really showed is that developers know so little that they think they know more than they do (an unfortunate example of the Dunning–Kruger effect). More than half of those respondents were not familiar with common software vulnerabilities, how to avoid them, and how they can be exploited. 92% said they needed more training on security frameworks, and 86% stated they found it challenging to practice secure coding. In short, they thought they knew enough, yet most knew almost nothing.

[evverx]

Hopefully that makes a dent in the need to improve maintainer experience.

I'm not sure how it can address the underlying issues most of which boil down to the fact that there don't seem to be actual maintainers anywhere at OpenSSF who can figure out how all that stuff affects actual maintainers. For example at some point the OSV database was introduced and somehow it turned into a machine helping to automatically file nonsensical CVEs and somehow I ended up on the receiving end of that nonsense. Promotional PRs fixing imaginary security issues wasted a lot of my time too. I wonder how that WG with basic training materials can change that?

[lucasgonze]

A project to address maintainer experience with Scorecard: https://github.com/ossf/scorecard/issues/3798

[evverx]

A project to address maintainer experience with Scorecard:

I think in terms of maintainer experience scorecard is more or less covered these days in the sense that there is a team at GOSST integrating scorecard into actual projects, talking to actual maintainers and receiving unabridged feedback. Issues they open are thoughtful and based on conversations with actual maintainers. I think it would be nice if those issues were prioritized instead.

[sevansdell]

@jkjell and I have taken over the reins of the Security Toolbelt SIG in the Best Practices WG, with ongoing support from @SecurityCRob, who helped jumpstart the SIG.

While we are not representative of the "Maintainer experience" we are interested to have our work contribute to and enable the maintainer experience within the broader OSSF community. Perhaps the links below, in conjunction with efforts of the DevRel community can help this conversation thread?

While we are still in the process of refining a roadmap and contributions, we have done quite a bit of work to date on the following topics, some of which support the maintainer experience:

• Personas
• Use cases
• Threat models of use cases to articulate risk
• Where OpenSSF capabilities can be applied to persona's use cases to address threats and reduce risk
• We would like to also --Identify gaps --Create patterns and templates making application of OpenSSF capabilities to reduce security risk in the software supply chain easier/faster/more efficient

[evverx]

Thank you for the links. As far as I can remember I took a look Security Toolbelt SIG and DevRel community some time ago. Security Toolbelt SIG looked too abstract to me and the DevRel community seems to have focused on conference talks. I'll try to take a closer look later.

--Identify gaps --Create patterns and templates making application of OpenSSF capabilities to reduce security risk in the software supply chain easier/faster/more efficient

Off the top of my head things like scorecard and allstar are reactive and can't be used to prevent, say, dangerous workflows that can compromise repositories from being merged. For example I'm not sure why people have to spend their time on looking for things that can be caught and remediated automatically when PRs are opened. It can't detect dangerous workflows where data flows have to be analyzed either and so on and so forth. All those things have been reported and ignored because it's much more involved to make those things work than to, say, generate markdown checklists.

I'll take a look at the personas, use cases and so on once I've gotten round to it. Thanks again for the links!

Edit: I'm sorry but I can't see how "Ursula the Upstream Maintainer" can affect anything. She is unlikely to attend OpenSSF meetings (and that's kind of sad because based on my observations all the decisions are made there so she is effectively excluded from the conversation regarding what sort of stuff is dumped onto her thanks to a bunch of automated tools). Generally I'm not sure how the following objective is going to be met

We need to ensure there's proactive, ongoing engagement with maintainers and their projects; "once and done" is too easy to get buried under the day to day demands".

given that even critical projects are currently ignored. To judge from https://docs.google.com/document/d/1hO6NuSiNr_7PO1QTYsB6qzcS8pAFW7p_6JT2y0XL5Nk/edit#heading=h.z9vrmtiy2usx the DevRel community should

Build and maintain relationships with the greater end-user and open-source communities.

but without attending a lot of meetings it's a one-way street anyway. If that sort of dynamics can be changed somehow maybe it can work.

Either way it's a lot to process so it might take a while.

[DianaOlympos]

Ok so i am going to try something slightly different here. I am going to explain what it is to be a maintainer first. This is ofc limited to my experience and knowledge of talking with others, subjective, anecdotal, etc. YMMV.

Then i will try to show based on this why the OSSF stuff does not help me.

Persona: Diana, the weekend warrior.

I maintain a couple of small packages and contribute new medium size but impactful features to my underlying ecosystem. Think a compiler optimisation for floats that takes a few months of work and extremely niche knowledge to get right. This is a really common and critical profile. I am in a loose network of other niche people doing the same in my ecosystem.

We usually get something like 2h per month to spend on FOSS. Sometimes up to 4h, sometimes less. 1 to 2h per month ate dedicated to simply updating base dependencies. This is when it is just a few patches or minor versions. Sometimes up to 4h are taken for this. Sometimes a big major version in an important dependency happens and it takes us 10h to fix, over a quarter.

This means that nearly all our time is spent handling dependency stuff. Basic stuff. Not security emergencies, or anything like that. Releasing a new version that just is kept up to date basically.

I do not have more time to give. Life is what it is, i have family, a job, friends, etc.

My tests are shit. I know it. I keep wanting to make them better. Fuzzing? I could not find the time to fix the bugs found anyway. Reading security oriented material? When?

Most of my time is spent fighting my build and dependency management tools. This is a constant hell, they break all the time in new obscure ways. I, by luck, keeps finding an incantation that makes them produce something that works. It uses non of the security stuff. I even go out of my way with my build file to deacticate all sandboxing, because it keeps breaking my builds. I suppose there are ways around it, but they would need me to understand how these things work and also to adapt them to my need.

I do not have the time. The basic tools breaks already too much and eat my time. Also wait. Tons of user reports!! But i only did a minor version update for that dependency which had an innocuous changelog?!?? Wait they messed up their versionning. This was API breaking change but they had no idea. I would have the same mistake. It is not obvious. Aaaargh. Well i guess i will have to tell everyone to use the old version for the next 6 months while we spend our 6h per quarter all fixing it. Hope no emergency security patch come through during that time.

At least the yubikey i have used for the past 7 years is used for my packages now... Oh no. Too old. Not supported. I guess i need to find money for a replacement now

Conclusion

There. You have it. Now, you want better security? Help me get some of the time fighting the tools back. Help me be able to install and build my software without needing shell hacks and root access to the whole machine and the internet. Help me just standardise the system root certificate store so i can access it. All these things that will give me the time to even think about what the OSSF talks about.

Edit note: there are some projects looking at this. Not from the security crowd, with highly limited funding. Rust as a whole is one, with tools there like semver-check but also just cargo. And other ecosystems too. On the other hand, autotools has no maintainers for a decade.

I understand the data you look at. The research you look at. The way you see the world. I understand why you think you are helping. And you are! At the margin.

But this is not the margins we need anymore. Stop looking at surveys. Sit down, live the life of the weekend warrior. Literally. Listen to them in person. Do qualitative work. And remember. Most weekend warrior are not on surveys. Not at conferences. Because their employers do not support them and they do not have the time and money to go there.

I will be, probably, at FOSDEM this year. Probably. In the EUC DevRoom. I say probably. It is in 2 weeks and idk how i will pay for travel or lodging there yet.

Plus it will take all my FOSS time budget for the year. Or i cut other important stuff in my life for it. And pay the prize. To hear again and again people touting how they are helping with all kind of work that does nothing for me.

But happy to talk more there!

[SecurityCRob]

Thank you @DianaOlympos for taking the time to put your thoughts into this thread. As always, they are very insightful, and I appreciate it. I will take your persona and merge it into our existing work within the toolbelt - https://github.com/ossf/toolbelt/tree/main/personas as your real-world perspective adds a lot of colour and value there. I will not be at FOSDEM, but I think several others within the Foundation would be, let me ask around and see and perhaps we could arrange a short talk between you and them. I'd be interested to hear more about which tools you and other maintainers commonly use and think through how we can influence things to help make them more reliable and invisible to your work. I completely understand the frustration of wrestling with systems/access/misbehaving tools/etc. The TAC is discussing this issue today (23Jan), and ideally we'll be able to get some folks to collaborate more actively on making this a priority for us.

[david-a-wheeler]

FYI, a few changes in the best practices badge project that I think might be relevant:

• We've made it easier to find "what's left to do". We have supported a mechanism to show this, but it appears some people didn't see it. PR https://github.com/coreinfrastructure/best-practices-badge/pull/2098 changed the name of the button to "Show only incomplete criteria", increased the color contrast so it's easier to read, and moved the button to make it more prominent.
• We've implemented a way to generate a markdown checklist, with marks on what is done and left to do. Once people are happy with it, the plan is to make that discoverable. More info is in issue https://github.com/coreinfrastructure/best-practices-badge/issues/2094

[evverx]

That's nice but personally my badge experience would be greatly improved if those badges were actually vetted and maintained. It's hard to get past dead links from 2017 and try to figure out why projects with, say, no fuzz targets anywhere (or links to their downstream fuzzing infrastructure or anything) meet the "routinely fuzzed" criterion with unit tests. I think they are great in terms of raising awareness but the "Consumers of the badge can quickly assess which FLOSS projects are following best practices" part is questionable.

Either way I was kind of hoping that DianaOlympos's comment (which is so well put that I don't even know what to say) could shift the discussion from badges and checklists.

[SecurityCRob]

I maintain a couple of small packages and contribute new medium size but impactful features to my underlying ecosystem. Think a compiler optimisation for floats that takes a few months of work and extremely niche knowledge to get right. This is a really common and critical profile. I am in a loose network of other niche people doing the same in my ecosystem.

We usually get something like 2h per month to spend on FOSS. Sometimes up to 4h, sometimes less. 1 to 2h per month ate dedicated to simply updating base dependencies. This is when it is just a few patches or minor versions. Sometimes up to 4h are taken for this. Sometimes a big major version in an important dependency happens and it takes us 10h to fix, over a quarter.

This means that nearly all our time is spent handling dependency stuff. Basic stuff. Not security emergencies, or anything like that. Releasing a new version that just is kept up to date basically.

I do not have more time to give. Life is what it is, i have family, a job, friends, etc.

My tests are shit. I know it. I keep wanting to make them better. Fuzzing? I could not find the time to fix the bugs found anyway. Reading security oriented material? When?

Most of my time is spent fighting my build and dependency management tools. This is a constant hell, they break all the time in new obscure ways. I, by luck, keeps finding an incantation that makes them produce something that works. It uses non of the security stuff. I even go out of my way with my build file to deacticate all sandboxing, because it keeps breaking my builds. I suppose there are ways around it, but they would need me to understand how these things work and also to adapt them to my need.

I do not have the time. The basic tools breaks already too much and eat my time. Also wait. Tons of user reports!! But i only did a minor version update for that dependency which had an innocuous changelog?!?? Wait they messed up their versionning. This was API breaking change but they had no idea. I would have the same mistake. It is not obvious. Aaaargh. Well i guess i will have to tell everyone to use the old version for the next 6 months while we spend our 6h per quarter all fixing it. Hope no emergency security patch come through during that time.

At least the yubikey i have used for the past 7 years is used for my packages now... Oh no. Too old. Not supported. I guess i need to find money for a replacement now

I've merged a first pass at your persona feedback @DianaOlympos . Can you please see if this meets what you were describing: https://github.com/ossf/toolbelt/blob/main/personas/README.md#name-diana-the-weekend-warrior The Toolbelt team has recently agreed that this Weekend Warrior persona will be one of the key focuses of their deliverables in the coming months (to streamline, simplify, and help integrate tasks, data, etc for this persona.

[DianaOlympos]

Will have a look later, if i have said nothing in a few days, feel free to ping again please

[webchick]

Today on the DevRel / Community chat, @Cheukting created a #maintainer-experience channel. :D The idea is for this to be where we direct folks we meet at conferences / meetups / etc. who have an interest in being consulted / included in initiatives from a "user experience" testing POV.

[webchick]

Also, I finally had time to catch up on this discussion, and +1000 to @DianaOlympos's extraordinarily accurate description of the vast majority of open source maintainers' experiences.

If I could pick a single "North Star" metric for the OpenSSF to focus on, it would be:

Time to Critical Project Maintainer Value == as infinitesimally tiny as possible

Given sufficient focus on that, not only would Weekend Warriors stand a fighting chance of learning about and adopting some of the great work that's gone into OpenSSF's various tools and resources, but there would also be beneficial "halo" effects of improving things for all other Open Source Maintainers, as well as the learners just dipping their toes into Open Source for the first time.

In practice, this means things like:

• Critical Project Maintainers should be able to understand in 2 mins or less what the heck it is $OPENSSF_THING does and what the benefit is to them to use it: https://github.com/ossf/DevRel-community/issues/35 is one approach we're looking at in the DevRel team.
• When incorporating an $OPENSSF_THING into their projects, it should be as dead simple as possible. 5 mins would be fantastic, anything over an hour of horsing around is too much. There was mention on the last DevRel meeting that OpenSSF Security Toolbelt is looking into a one-click button type of thing for maintainers that (I believe) leverages GitHub Actions for automating a bunch of manual stuff. This Is The Way.™ ❤️
• Given the extremely valid statistics mentioned by @david-a-wheeler above, all of these $OPENSSF_THINGs should absolutely not assume a baseline understanding of really any security best practices whatsoever among Critical Project Maintainers, and should instead go out of our way to either avoid jargon, or define it clearly when it's unavoidable, and also to link copiously off to resources to learn more about those things. Having to stop to Google / ChatGPT / etc. things also chips away at Time to Critical Project Maintainer Value.
• <Your Ideas Here!> :D

I've been super impressed here btw with both open source maintainers' willingness to share "real world" feedback with the OpenSSF, and also with the OpenSSF's willingness to engage with some tough comments and provide additional context. We're all ultimately on the same "side" here and most of the comments here have really shown that. Great job, everyone. 🎉

[evverx]

a #maintainer-experience channel

For various reasons some people (myself included) can't use Slack (and most of the communication channels OpenSSF uses) but I'm glad that folks are going to be met there by people who can open thoughtful issues like this one. It would be great if the DevRel team could convince TIs/WGs to use asynchronous means of communication like issues on GitHub and commit message a bit more. Some projects like OSV are great at posting updates but some projects refuse for whatever reason even when asked explicitly and move discussions to meetings.

Critical Project Maintainers should be able to understand in 2 mins or less what the heck it is $OPENSSF_THING does and what the benefit is to them to use it

I agree it would be great but I think it would be nice if TIs/WGs did some sort of research first. One example would be https://github.com/ossf/sbom-everywhere/pull/27/files#diff-84d5c6f22596cd94bba61aea654d191da533d7016ef2682e2805ddcbdfefc397R11. I don't know if that plan went anywhere but at least it shows that that WG is aware that there are a lot of different ecosystems, a lot of different ways to distribute software and is interested in actually doing things before giving out untested guidance. They also talk to people who have been doing those things long before 2021 so they know where it doesn't make any sense to shift those downstream responsibilities to upstream open-source projects for example.

[sevansdell]

FYI https://github.com/ossf/tac/issues/330 to make getting/staying involved in TIs easier.

[sevansdell]

@webchick: now that https://github.com/ossf/tac/issues/330 to make getting/staying involved in TIs easier. is open, should we make sure these comments are carried forward in that issue and bring this one to a close. There has been really great discussion here. If we keep it open, what work remains to help close it out in your opinion?

[riaankleinhans]

/vote

[riaankleinhans]

Gitvote was added as a tool to test for stream lining the TI Funding process. The members of the GH group "TAC" can vote by commenting with an +1. -1 or eye on the Gitvote block in this issue. Until the TAC is satisfied with the process the GitVote outcome would not be binding.

Community members can show their support by also voting, however only the "TAC" GH Group's votes will count.

The current passing threshold is 70% and the committee is the TAG GH group. The vote say open fo 6 week and an announcement is sent on the GH/TAC/Discussion

All these parameters can by fine tuned or changed here Please reach out if you have any questions.

[git-vote[bot]]

Vote status

So far 0.00% of the users with binding vote are in favor (passing threshold: 70%).

Summary

In favor	Against	Abstain	Not voted
0	0	0	9

Binding votes (0)

User	Vote	Timestamp
@steiza	Pending
@torgo	Pending
@mlieberman85	Pending
@bobcallaway	Pending
@lehors	Pending
@SecurityCRob	Pending
@marcelamelara	Pending
@camaleon2016	Pending
@sevansdell	Pending

[sevansdell]

@riaankleinhans I don't think we need a vote on this issue. I have greatly appreciated the dialog here. @SecurityCRob, now that you are the Chief Architect, and I know there are persona conversations related to this issue on your plate, how should we proceed to close out this issue? Is an acceptable solution to work through your role and partnership with TAC/DevRel Community? We can close out this issue, and use it as a reference for future dialog. @webchick with you being an initial contributor, how do you feel about this issue disposition?

[evverx]

I don't think anything has substantially changed. OpenSSF is still focused on consumers and that xz response didn't help to win hearts and minds of maintainers either.

The things discussed here were either archived or are on their way to target companies:

• The toolbelt initiative didn't go far: https://github.com/ossf/toolbelt/issues/14
• The 201 security architecture course isn't going to be publicly available. At least according to the Best Practices meeting notes:

  The plan will be to charge for this course and use the proceeds to help fund the OpenSSF

and the list goes on. On the bright side https://www.sovereigntechfund.de/ seems to participate in some meetings so maybe they can help by bringing their experience.

[riaankleinhans]

/close-vote

[sevansdell]

I put a link to this issue in a comment for MVSR v3 discussion.