Health of the Notary V2 project

[JustinCappos]

As I understand it, the TOC is starting to review projects with a consideration to reassess their level in the CNCF or even to remove them altogether. I wanted to bring the Notary V2 project to the TOC's attention as a project that is misplaced and worthy of review.

First of all, the original Notary V1 project was added by the CNCF and was voted in both because it had a strong security foundation and a substantial user base.

Strangely, the Notary V2 project has none of the original Notary project members, none of the lines of code from Notary V1, and none of the security design. It is effectively a completely different project that has taken the same name in order to preserve the incubating status in the CNCF. Even worse, it is at incubation level and making use of CNCF resources / marketing / reputation, yet has had no security reviews, etc.

I would kindly suggest that the TOC consider either removing Notary V2 from the CNCF or asking it to reapply to the CNCF.

Notary V1 (the original) likely could also plausibly be archived or reviewed at some point, but this is of less urgency as it did actually receive due diligence at some point.

I know I raised the same concern back in July 2021, but after talking with others in the community I thought it was worth raising again. As transparency is an important part of open source foundations and projects, after raising this issue a week ago to the TOC privately, I am now making this request public.

I am including below the 2021 email where this issue was raised to the TOC. I will note that at time we were hoping for governance changes that would enable us to participate and prevent the project from making repeated, obvious security errors. Now, while to the project's credit, it's website does not claim the project has or provides any security properties, its use of the Notary name and CNCF Incubation status is trying to build off of the Notary / CNCF brand, without any review.

The relevant email is included below.

To: The CNCF Technical Oversight Committee (TOC)

Date: July 2 2021

Subject: Concerns about the governance, security, and branding of the Notary v2 project

We, the signatories named below, have been contributing to or closely following the Notary v2 project ever since its inception in late 2019. Notary v2 was first created to resolve some usability challenges with Notary v1, while maintaining its security guarantees, which are based on The Update Framework (TUF).

Unfortunately, we have seen the Notary V2 project stray away from these original goals, due to problems with its governance. We are now growing increasingly alarmed, and have grave concerns with the v2 project as follows:

A lack of open development. Several maintainers of the project have frequent private meetings with corporate members of the community, while leaving the open source members in the dark. This has led to the creation of a reference implementation that does not meet the stated goals of the project, especially the security goals.

A lack of designing for security. For example, there is no rigorous threat model for the project. This is especially concerning since this effort must be robust against attack, especially given recent events [1, 2].

Use of the Notary name and brand in an entirely separate project. Proposed requirements, specifications, and implementations of Notary v2 are not based on the Notary v1 codebase and do not plan to be compatible with Notary v1. The project is essentially completely distinct from the original Notary project.

Notary v2 has significantly moved away from the security guarantees of v1. The guarantees of v1 align closely with TUF, and the two projects are widely known to be related. A new Notary v2 project that does not achieve these security goals may mislead others that the project provides strong security as this was the differentiator of Notary v1.

Although we have repeatedly brought these concerns to the attention of the Notary v2 project over the past six months, we have thus far not seen any progress toward resolution [3, 4, 5, 6, 7, 8, 9].

As Notary v2 moves forward as a CNCF project, we strongly recommend:

Following a clear, transparent, community-driven governance system

A clear separation of the Notary v1 and v2 projects

Determining the future of Notary v1 as a separate project

Moving of v2 to sandbox to encourage a more detailed review of the project

Renaming of v2 unless it maintains and improves the security guarantees of v1

Threat modeling and security audits/reviews in line with the CNCF requirements for sandboxed projects

We ask for the CNCF TOC to help us to resolve these concerns. We love to discuss this further before opening this up to the broader community.

Signatories

Justin Cappos (CNCF TAG Security Tech Lead, TUF, in-toto, Uptane)

Jason Hall (sigstore)

Luke Hinds (sigstore)

Trishank Karthik Kuppusamy (TUF, Uptane, in-toto, CNAB Security)

Dan Lorenc (sigstore)

Marina Moore (TUF, Uptane, sigstore)

[dims]

@JustinCappos we do it two steps, we gather info, make a decision and then open another issue for archiving if at all. Let's use this to gather thoughts from folks please.

@amye can you please fix the title of issue to "Health of Notary v2" project, remove the archive tag and check list. thanks!

[JustinCappos]

@JustinCappos we do it two steps, we gather info, make a decision and then open another issue for archiving if at all. Let's use this to gather thoughts from folks please.

@amye can you please fix the title of issue to "Health of Notary v2" project, remove the archive tag and check list. thanks!

I've addressed the naming and checklist parts of this.

[dlorenc]

Hey,

I wanted to share some experiences working with the Notary v2 project.

Disclaimer: now I maintain cosign and many parts of sigstore, which are also LF Projects currently part of the OpenSSF. These are commonly perceived to compete with Notary v2.

At the end of 2020, I, along with many others, tried to join the Notary v2 community and contribute to the project. I found the project impossible to navigate and very unwelcoming. After months of not being able to make any progress and watching PRs from contributors linger, I tried to research the structure of the project. I was unable to find any information on who was allowed to merge code, how the project was governed, or how folks were supposed to contribute, and I filed a bug asking for this clarification: https://github.com/notaryproject/notaryproject/issues/55.

Many concerned folks, including some from TAG-Security, wrote the letter Justin attached here to the TOC over a year ago, and the governance issue was finally addressed after about 6 months, governance was put into place, and the project was restructured. This all happened behind closed doors and many regular contributors were not invited to participate.

The original Notary v1 project is basically archived at this point, and the Notary v2 effort has been dragging on for many years. To Justin's point, security feedback from members of TAG-Security has basically been ignored, and I don't think either project (v1 or v2) is in a state the CNCF should feel comfortable recommending.

Some transition period is reasonable between major versions of a project like this, but the time scale here is leading to lots of confusion among the community. I don't believe either version of the project qualifies as an Incubating project in the CNCF today, and I believe it is time the TOC or TAG-Security take action to clarify this to the community.

[imjasonh]

I was a signatory on the original July 2021 letter, when I was at Red Hat, along with the other signatories @lukehinds @trishankatdatadog and @mnm678. Thanks @JustinCappos for bringing this up again, since nothing substantive seems to have happened since then.

IMO Notary v2 trades on the reputation of Notary v1's name and CNCF project status while otherwise changing all other aspects of the project's implementation, security design, and maintainers. This is a dangerous precedent, and a bad look for the CNCF, but I think it's actually relatively straightforward to resolve, by asking Notary v2 to find a new name and go through the CNCF Sandbox process like any other new project, as @JustinCappos suggested.

[lukaszgryglicki]

DevStats updated to track the new org, details here: https://github.com/cncf/devstats/issues/382#issuecomment-1353502615

[SantiagoTorres]

Disclaimer: I was a TUF core contributor up until a couple of years ago. I gave some talks on Notary V1 on one or two KubeCons as well.

I'm incredibly sad to see that Notary has followed this path.

I a lot of points have been made but, as somebody who spent near to two years trying to steer things to a better direction in this community, I want to give my perspective. In particular, I want to make sure people know how the lack of process and adherence to open source software has led us to where we are.

First off, the mention of a lack of security review issue has been raised, but there is very little talk about how it was discarded. My impression was that the newcomers didn't attempt to understand any part of the previous design and, instead, wanted to start with a blank slate. The rationale was never developed, other than "we want to use this instead" and any explanation was discarded right away. I remember pointing out that we had security design reviews by top security consultants multiple times to no avail. I don't want to speculate too much, but from the comments in the calls I get the impression that new security design decisions were done behind closed doors and apparently by an edict of some people inside a corporation that did not even bother joining any calls.

Further, as @JustinCappos says, this renders all previous security design reviews and security analysis by both academic researchers and industry researchers (who were paid for by the CNCF) moot. I don't understand how this is admissible by the CNCF without a clear rationale. I believe the CNCF has a goal to have open, respectful and constructive discussions.

More concerning is there was no explanation as to why this had to change or why the new design decisions were made. Anybody from the community had to either get in line or be yelled at/ridiculed/avoided so that things weren't either discussed or considered "past discussion". I think this is quite the affront to the philosophy behind not only the CNCF or the LF, but Free and Open Source Software as a whole.

I think this is something to highlight. These newcomers weren't joining a community, but taking over it. I'm not entirely sure why the governance had to be made by them. I'm still not sure why the notary repository was moved away from the TUF org into a new one controlled by them. I don't think any single member of the community was consulted about this.

I took personal concern of seeing this meeting. My concern is based off of:

1. This meeting was apparently not in the calendar for Notary V2
2. On this timestamp, I see and hear a Microsoft PM pushing a group of AWS Engineers to commit to a design on the word of some Docker Executive that was not present saying that Docker are committing to something.
3. I don't think there were any people not from MS or Amazon on this call.
4. They discuss launch dates for their corporate products based off of Notary V2 (I think?). I don't think this is how open source is released.
5. They somehow can't help but to talk about me and my critiques in a dismissive way.
6. I don't know how to interpret the sentence: "I don't care about Notary, I care about our Customers"

It may be my bias, but this definitely does not feel like an open source software project meeting. For the record, I have no qualms with companies trying to align requirements to their interests, for as long as they do this in the open.

This is a central part of my concern, which goes beyond the technicalities. I don't believe in the prospects of notary being a community following open source philosophies, and thus it should not be endorsed or hosted under the CNCF or the LF umbrella without serious review on the status quo and all the pathologies that led to this.

[trishankkarthik]

I am one of the original signatories of the letter dated July 7th 2021, and I would like to now also add my support for:

1. Either removing Notary v2 as a CNCF project or asking it to reapply (although I’m strongly inclined towards the former given its longstanding behavior).
2. Archiving Notary v1 once and for all (for security reasons as it no longer sees active development).

My reasons for revisiting the membership of Notary v2 in the CNCF include:

• Poor security guarantees (even compared to Notary v1). There should be a formal threat model along with comparisons to other security technologies.
• Poor collaboration. I was involved from the end of 2019 till mid 2021 IIRC, when I had to leave because the project was simply not interested in incorporating our feedback despite promising open collaboration. In particular, the project kept ignoring and sandboxing @mnm678’s hard work and research.
• Bad governance. Some of the maintainers hardly showed up at all during the meetings I attended, and yet represent the project in public talks held annually or so.
• Slow, closed development. Despite years of effort, there were not even public beta, never mind RC, releases until recently.

For these reasons and more, it seems to me that unfortunately Notary v2 does not reflect the best practices of a CNCF project, and as such, its membership should be revisited. Thanks for your consideration.

P.S. These are my personal views and do not reflect those of my employer. Disclaimer(s): I am a TUF/Uptane/in-toto/Sigstore contributor.

[mattfarina]

The TOC encourages community members to provide context and constructive and community building recommendations and information around Notary v1 and v2, however we are not going to discuss this until January (meeting to be determined). We appreciate everyone’s passion on this topic and look forward to continuing it in the New Year after the holiday break.

[dims]

Summary of email from Liz last time when the Notary question popped up

Context:

• Email with questions/concerns from the community (Dated Jul 2, 2021) sent to the cncf-private-toc mailing list were:
  • Justin Cappos (CNCF TAG Security Tech Lead, TUF, in-toto, Uptane)
  • Jason Hall (sigstore)
  • Luke Hinds (sigstore)
  • Trishank Karthik Kuppusamy (TUF, Uptane, in-toto, CNAB Security)
  • Dan Lorenc (sigstore)
  • Marina Moore (TUF, Uptane, sigstore)
• Response Email from Liz Rice (then Chair of TOC) summarizing the discussion and guidance from TOC (Jul 16,2021)

Guidance:

Notary and TUF are separate projects, but this is not very clear because the Notary v1 repo currently[1] lives under the TUF organization on GitHub. As a first step the TOC will urge the maintainers to 

- move this project to the Notary organization as soon as possible 
- refresh the maintainers list which is evidently stale
- provide clear guidance to end users as to the status of the project - namely, that v1 is an implementation that can provide signing functionality, but that the project is now focused on a new implementation that does not follow exactly the same specification, but aims to provide a more easily used signing implementation. 

Resolving this would address many of the concerns you raise as it would clarify that the projects are decoupled (and that Notary is not necessarily bound to being compliant with TUF if that no longer meets the goals of the Notary project).

[1] https://github.com/theupdateframework/notary
[2] https://github.com/notaryproject

Additional Notes:

• Regarding the a clear, transparent, community-driven governance system question:

• We’re aware that the Notary project does have an active community but that it needs to clarify and codify the project governance as a matter of urgency. Our understanding is that this is in progress; we will set a deadline that this should be resolved by the end of August.

• Responding to clear separation of the Notary v1 and v2 projects:

  
  The TOC doesn’t believe it will serve the wider community to follow this recommendation for the following reasons:

• We believe that the Notary name is understood as a project that addresses the need for securely signing artifacts.

• There is precedent for re-implementing projects without requiring them to reset into the Sandbox (for example Fluent / Fluent Bit, Linkerd v1/v2)

• Notary clearly falls into the Cloud Native landscape and has participants from multiple organizations in the CNCF Community, so we want to encourage this collaboration

• A significant proportion of adoption is through other projects like Harbor, which should ease the transition for end users.

The alternative approach would be to archive the Notary project and encourage “Notary v2” to reapply with a different name, but we believe this would create more confusion for end users.

- Responding to `Threat modeling and security audits/reviews` and `behind closed doors`:

• Notary is of course subject to the same requirements as any other incubation project. The security requirements at this level require security processes to be in place but we don’t currently have any requirements around threat modelling (though we do encourage projects to collaborate with TAG Security on their security assessment process).

• We also note the concern raised about development “behind closed doors” and encourage members of the project to make the TOC aware if decision-making happens in a private fashion. However, we also note that collaborating on a project together does not preclude people from having private discussions - for example, it is acceptable to solicit feedback from a smaller group before opening something up for broader input. It is the decision-making that needs to be open.

• Noting your intention to discuss these concerns at the next Notary meeting, we propose to publish this thread on the TOC mailing list, and add this to the agenda for discussion at an open TOC meeting in August, where we would be delighted to have you join us for the discussion (and this would allow time for the Notary meeting to take place first). Does that seem reasonable to you?

[JustinCappos]

@dims Thanks the recapping the TOC's handling of this issue last time.

Note, that apart from us sending the letter, the signatories received no response except this (private) message. I am not aware of any efforts to reach out to any of us for further details / discussion or any follow up to check with any parties about their concerns.

One of the most frustrating aspects of the past decision was it being worked behind closed doors. These sort of private, backroom dealings are one of the hallmarks of the problems with the Notary-v2 project. They also go counter to several of the CNCF's key principles laid out in the charter (2a, 3b, etc.).

As for our current request, this was brought up last year in the notary-v2 slack, it's clear they are aware of this request and have chosen not to comment publicly.

As nothing is happening in an open manner, it seems there is some combination of: 1) the Notary-v2 project members do not feel there is a convincing public rationale for the project not to be re-evaluated / removed 2) they are trying to work this issue through non-public channels.

Regardless of the outcome, we would encourage the TOC to ensure the process plays out in a public, open manner.

[dims]

Justin,

Just to be clear, a few of you sent an email to the private toc mailing list and the TOC responded via the same channel. Any follows up it is implicit to keep to the same channel of communication. Let's not confuse this pattern of communication with the general goings-on. TOC does this to ensure we don't step on folks who are trying to get a bad situation under control and add fuel to the fire. Just to be clear, TOC hasn't had any communication from anyone AFAIK on this $TOPIC after the last 2 emails from you and Trishank responding to Liz's email.

We are happy to get this on the TOC agenda in an upcoming public meeting.

@amye can you please check when we can do this so we can inform everyone and ensure folks are represented?

[amye]

Meeting is January 17th

[dims]

Here's how to add it to your calendar: go to https://www.cncf.io/calendar/ search for TOC or go straight to for Jan 17th and you should see this.

[anoncam]

is it dead yet. i have no desire to dance around an issue that creates confusion and unnecessary division of consensus on what is obvious: sigstore is our future. Who must be pestered to the point of a real debate to close this chapter @dlorenc

[dims]

@anoncam I'd recommend reading https://github.com/cncf/toc/blob/main/PRINCIPLES.md and focus your energies on where you think you should.

[dims]

Folks,

Thanks for folks who made it to the meeting today, the recording from today is here: https://youtu.be/ODlZduZo32U

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

[trishankatdatadog]

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

Thank you, and sorry I couldn't make it yesterday—my flight was rescheduled. Happy to meet offline if interested. Thanks for looking into this!

[mlieberman85]

Folks,

Thanks for folks who made it to the meeting today, the recording from today is here: https://youtu.be/ODlZduZo32U

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

What is the timeline on this? There has been some back and forth on this general topic for a while and though I agree with the need to get more feedback from the community we want to ensure that this gets lost in the mix.

Similarly, is there a process for this sort of situation more generally?

[dims]

@trishankatdatadog can you please ping me on slack, we can set up a time to chat? (dims on both k8s and cncf slack). thanks!

[TheFoxAtWork]

Folks - the TOC is actively working on a resolution and we expect it to be finalized soon. We’ve had many community members reach out to us to voice concerns, experience, and perspectives both publicly and privately. We’ve even reached out directly to a based on previous discussions. Community members are welcome to reach out to TOC members to share, and the TOC will review these to determine if they will influence resolution however we may not respond to every DM or email. Please bear with us as there are many factors to consider that are unique to this project but also considerate of all projects and their governance.

[mattfarina]

We want to start by thanking everyone that we, the TOC, spoke with while reviewing this situation. It helped provide some much needed context.

We also ask all parties to be empathetic towards one another when facing technical conflict. We want to welcome other perspectives and experiences when evaluating community concerns.

The TOC has met numerous times at various intervals with many community members passionate about Notary to understand this issue and have also looked outside of the CNCF to how other foundations or projects have navigated similar concerns. We have heavily considered the charter we have and the goals we maintain for projects and you will see elements of this below.

Guidance given to the Notary project in the rest of this response is the type of feedback often given during the due diligence process during an incubation and graduation review. Often changes need to be made during the incubation or graduation review prior to a vote.

---

Action Summary For Notary:

The following is an action based summary for the Notary project. Context around these and details on reported concerns not addressed here are below the summary.

• The TOC expects projects to be self-governing, this means community members abide by their governance or make motions to propose, discuss, decide, and document changes. We expect Notary to adhere to their governance and request the project consult with TAG Contributor Strategy for guidance on this.
• Notary needs to refresh the documentation around all its subprojects to eliminate confusion by it’s adopters and contributors (past and present). This may include archiving stale repos but it is up to the project to follow their governance process in executing this.
• Notary and its sub-projects will undergo a security audit targeting the v2/notation work.
• Suggestions for specific improvements of Notary should be made to Notary in accordance with their contributor guidelines and governance.

---

Technical conflict in projects should be documented by the project as part of their self-governance. It should consider all perspectives and move to a vote under the specified governance process at the time the conflict occurs. Should the governance be lacking, the project should remediate the governance first and then return to technical conflict resolution. Should this not solve the issue, the project may escalate to the TOC for technical remediation through a to be determined process. Escalating to the TOC is not a path for community members to have the TOC direct a project to change direction when a reasonable governance was followed.

Notary has a documented governance and it, with the process above, should be followed in this technical conflict resolution.

Note, after the quoted email above was written the Notary governance was updated to account for sub-projects like Notation (in this thread referred to as Notary v2).

Formal governance is designed to introduce structure and consistency. We highly recommend all projects lean towards formal governance with flexibility to adjust. Having and following a governance is a graduation requirement and in the graduation process all projects have their governance reviewed. Having a governance reviewed by TAG Contributor Strategy prior to going through the graduation process can help make the graduation process, which often includes changes, easier.

---

We fully expect projects to refresh their goals and objectives and to document those proposals and the corresponding decision. Changes in goals and objectives can lead to changes in design.

CNCF projects have a rich history with sub-projects. The TOC is not making changes in this area. Sub-projects should have their maturity clearly documented in relation to the primary project, and not be assumed to be at the same maturity level..

The TOC finds no issue with Notary having sub-projects or changes in design. The maturity of sub-projects with relation to Notary does need to be clearly documented. Changes in design have precedent of happening during the Incubation phase.

Projects may reasonably expect divergence in naming. When this occurs it is up to the project to make a documented decision with reasoning so that contributors and adopters can easily navigate each element of the project and its use. This includes cleaning up heritage or legacy naming conventions and documentation within the community while also archiving the appropriate repositories.

Notary, Notary v2, and Notation are all discussed as part of the project. There is the Notary v1 codebase, the Notary v2 specification, and Notation which is a reference implementation of the v2 specification. Clarification is needed to help contributors and end-users of the project. Questions arise such as:

• Is Notary v2 the general Notary specification and Notation a reference implementation? If so, why is the v2 specification versioned as a v1 and versioned like Notation? Or is the Notary v2 specification the specification for the Notation sub-project?
• Which of the sub-projects own the Notary v2 specification? Is it Notation, Notary, or an unlisted one? This may help clear up the first question.
• How are end-users supposed to view Notary v1 and v2?

We appreciate that keeping external documentation clear, consistent, and up to date can, at times, be a challenge. It may be useful to have a community manager to help keep the communication clear and consistent.

The documentation for the Notary project needs changes to improve clarity.

---

Projects, particularly security projects, should plan to go beyond minimums as a matter of building trust with adopters. For instance, achieving an OpenSSF Best Practices Silver or Gold badge. Moving beyond the passing badge requires documentation of elements such as an assurance case.

Given that the specification and Notation are at a major version release candidate stage, this is an ideal time for a 3rd party security audit of the specification and implementation. We will work with the project to get this process started.

---

Community members that have issues with the openness and transparency in which the project conducts their collaboration and development activities should discuss these with the project for resolution. Often technical challenges with the tools used may be the cause and this is not immediately apparent to community members.

Some ways we observed Notary currently operating with openness and transparency include:

• The Notary project currently has a regular public meeting that is documented in the project and listed on the CNCF calendar. An agenda and meeting minutes are captured in a public document.
• Development is managed through issues on the GitHub repositories and a project board.
• Notary is leveraging CNCF slack to hold online conversations.

Suggestions for specific improvements for the Notary project related to openness and transparency should be directed at the Notary project.

---

If the Notary project has questions about the guidance from the TOC, we are happy to provide more detail.

[JustinCappos]

Thanks to @mattfarina and the rest of the TOC for their assistance here.

We agree that this project is in need of a security design review and we appreciate the TOC's taking steps to remedy this problem. (I've not seen many successful security projects do a design review only after reaching a major release candidate stage.)

The clarification around naming / name use is also addressing a major concern, that even spills over to questions we get in the TUF project. It would be great to have this clarified as well.

I'm hoping these issues will be resolved and all can move productively forward. What is the timeline the TOC is requesting for these actions?

[trishankatdatadog]

@trishankatdatadog can you please ping me on slack, we can set up a time to chat? (dims on both k8s and cncf slack). thanks!

Just for the record, I want to thank @dims for taking the time to hear my concerns! Hoping we can all resolve this soon enough.

[mattfarina]

What is the timeline the TOC is requesting for these actions?

We are asking that the security audit be queued up now. It takes a little bit to get this started and involves the CNCF staff and a third party. I would like this to happen as quick as possible within the CNCF existing processes for this.

The other parts we have not put a timeline on. Dictating a timeline can be difficult. You need to know a projects velocity, available capacity, and other factors so you can make sure you get the necessary changes without putting undue burden on the people working on the project.

We will be keeping an eye on the Notary project to ensure they continue to make progress. If progress stalls we will work with them to ensure things get back on track.

[anvega]

I, for one, find the resolution dissatisfactory. The proposed actions fails to address the fundamental concern. It's kicking the can down the road and prolonging discord. The issue remains that of a sub-project perceived as rogue misappropriating the name of the main project and circumventing due diligence. If a subproject diverges in goals and objectives from those of the originating project, the project should be emancipated from the originating project organization.

We are talking the equivalence of a set of individuals rebooting the TOC and starting a TOCv2 without the consensus of seating members, and then excluding the participation of those same members as original TOC attempts in futility to course correct to its true technical charter and mission. The implications are serious. Due to the disconnect and failings in reparation, those members eventually gave up on recourse and started a new project outside of the CNCF which give birth to a separate foundation as they found the committee chartered with technical oversight was unwilling to step in and exert technical oversight. Said project has since superseded notation in traction, growth, and maturity.

For what it's worth, I do think Notation deserves an earnest shot of standing up on its own merit. I'm sure the toll of discord has been as draining on them as it has on those with opposing views. Having clear separation and delineation of two projects should do justice to what is clearly two separate efforts and workstreams by two distinctly different group of folks at two different points in time. Conflating the two under the same name and versioning is what has allowed confusion to seep into this.

With the spirit of reconciliation at heart, the desired outcome for all involved is for an end of the v1 and v2 discourse. Simply Notary and Notation moving forward.

As such, I propose the following course of prompt action:

1. Rename this issue "Health of the Notation Project"

2. Cease using the name v2 in project repository and all other assets with a set deadline. Ideally this should happen as soon as possible. Larger renaming efforts have been conducted foundation-wide in a shorter span of time than this issue has lingered on for(see Inclusive Naming Initiative, SIGs to TAGs). Project assets are under administrative control of CNCF Staff and these changes can be performed by such personnel without undue burden on the people working on the project

3. Rename project website from notaryproject.dev to notationproject.dev

4. Recategorize Notation as a standalone project

5. Level Notation to sandbox

Asking we can collectively exhibit ownership by stepping up and doing better. Eager we can call this over with so that we can get back to actually doing open source work.

[lachie83]

Action Summary For Notary:

The following is an action based summary for the Notary project. Context around these and details on reported concerns not addressed here are below the summary.

• The TOC expects projects to be self-governing, this means community members abide by their governance or make motions to propose, discuss, decide, and document changes. We expect Notary to adhere to their governance and request the project consult with TAG Contributor Strategy for guidance on this.
• Notary needs to refresh the documentation around all its subprojects to eliminate confusion by it’s adopters and contributors (past and present). This may include archiving stale repos but it is up to the project to follow their governance process in executing this.
• Notary and its sub-projects will undergo a security audit targeting the v2/notation work.
• Suggestions for specific improvements of Notary should be made to Notary in accordance with their contributor guidelines and governance.

Thanks @mattfarina and TOC. As a member of the Notary community, I wanted to acknowledge that I have seen the action summary above and am working with the community on each of the items. We will continue to link work item issues for tracking.

[dims]

I, for one, find the resolution dissatisfactory.

@anvega thanks for the input. The TOC has been deliberate, thoughtful and has set a course of action. please commit to it. Please switch the mode to looking forward and see how we can move forward.

Is there anyone else who hasn't spoken their piece? This is your chance.

[dlorenc]

Is there anyone else who hasn't spoken their piece? This is your chance.

FWIW I think this statement on behalf of the TOC would be stronger without this sentence. It comes off as minimizing and dismissive.

[dims]

@dlorenc i said this same thing before and ended up talking to @trishankatdatadog ( https://github.com/cncf/toc/issues/981#issuecomment-1416311317 ) Trishank pinged Santiago & Marina, but we haven't heard from them. hence the call for any more feedback. Apologies.

[anvega]

@anvega thanks for the input. The TOC has been deliberate, thoughtful and has set a course of action. please commit to it. Please switch the mode to looking forward and see how we can move forward.

For the record, I have no personal stake one way or the other. I simply see an omission in proceedings and failure of duty which I’m obligated to flag.

One one hand, there is one group of people who tried to overcome the perceived limitations of the projects and felt they were working on the next version and found it suitably convenient to do so under the same org.

On the other hand, there are set of individuals who strongly feel they were wronged and ostracized from the project, and something shouldn’t be called what is called or lived where it does. Much less that other new project that had nothing to do with them reap the rewards and status for the work they did all by themselves on the first project to get there.

If the contention is versioning, why not simply split the two and give both the space they deserve? Rebooting would allow folks on one end to come to terms and others to process and move on.

There is an untold story here of marginalization and ostracism which is important to surface. Among a group of predominantly BigCo backed players, students was singled out and their voices and contributions were suppressed by one actor. They dedicated 18 months of their dissertations which were effectively trampled and down the drain for what transpired. Students are told to emulate behaviors of professionals in the industry but instead were left subject to treatment that is unacceptable in a professional environment. Despite all of that, they hung around “chopping wood and carrying water” for which I'm stepping up in support. We can do better than this and we owe that to them.

What's been said here instead by the only meaningful proposed change being adding a 'caveat to docs' is forcing these individuals to relive the indignity they were exposed to. Relive the adversity they were routinely confronted by at the time. The name and where the project sits is an affront to them and those of us that know what they had to deal with. Furthermore, the inaction also dissuades academia from further participation in open source.

If the TOC seriously believes it's thoughtful and deliberate in the the message you are sending, I ask you to think again. What is it that truly matters? Giving up a name or giving up on people?

[dims]

@anvega there are community process(es) around code of conduct. please ask the folks to avail of that. yes, we have been hearing "lines have been crossed", clearly if things are they are characterized, I'd request folks to use the processes for the problems. I definitely want a resolution, NOT from the TOC, but from CoCC. Let's stop casting aspersions here when it clearly falls under their jurisdiction. This is what the interim CoCC is for. Please follow up with the students and guide them to the right people who are tasked to do that.

Clearly i wish to thank the students for sticking around and doing the right thing.

[anvega]

This remains a complaint pertinent to the purvey of the TOC. The Code of Conduct Committee is not in a position to rectify as it doesn't deal with project procedural matters. The TOC is the body chartered with "architecture for the projects, aligning projects, removing or archiving projects" I instead invite you to reconsider the TOC's latest pronouncement and reformulating a path forward in the light of new information.

To paraphrase @JustinCappos : Complaints and criticisms are equally valid before, then, and again now. None of it has been addressed in a way that the complainants feel is satisfactory. In fact, it’s almost entirely opposite to what is fair, equitable, and reasonable.

Those words were expressed in the same recorded meeting for which there is a link to on this issue and I reiterate them again. Rather than internalizing and integrating the feedback then and now, there is an immediate defensive argumentation for a predetermined decision despite the portrayal of holding space to listen. Not an aspersion but the TOC stance is seen by the complainants and myself as overtly intransigent, even exculpatory. Shunning.

To recap: A new project which has done harm even when that project shares a spec should not ride on the coattails of an existing project. Clear non-adherence with Charter Section 9(e) and the Do Not Harm TOC principle:

Charter Section 9(e): New open source projects initiated in CNCF shall complete a project proposal template adopted by the TOC and be approved by the TOC for inclusion in CNCF. The TOC members shall be afforded sufficient time to discuss and review new project proposals. New project proposals shall include details of the roles in the project, the governance proposed for the project and identify alignment with CNCF’s role and values

Do no harm, by enabling robust, well engineered projects and supportive processes to be leveraged for greatest adoption likelihood.

The principle you quote of "Disagreeing and Committing" is characterized in it's own definition as "quite harmful" after a decision has been made[1]. Disagreeing and commiting as at odds with the very "neutral consensus" that the TOC charter stipulates:

Conflict within a team is highly beneficial during the early stages of decision making as it helps the team to arrive at a better solution. However, conflict that persists after a decision has been made is quite harmful and prevents the team from executing effectively

Aligned with the Code of Conduct, I hope the TOC reciprocates demonstrating empathy and kindness, being respectful of differing opinions, and gracefully accepting feedback. Given we are a free thinking community sharing a common platform, I hope to continue to use this forum without getting told to politely shush by "switching mode" or that I had my chance to speak up or that others only get one.

Ultimately, if the TOC recuses itself from providing a satisfactory resolution to the complainants, can you point us to the procedure to follow to further escalate? Should the case be plead to the Governing Board next? Worth noting the complaints for the exception of one Platinum Member, are all Non-profit, Silver, and Gold Members of The Foundation.

[dims]

@anvega At any time for any reason you can talk to these folks:

• You can talk to Richard Hartmann ( Developer Seat Rep on the GB https://www.cncf.io/people/governing-board/ )
• You can talk to Arun Gupta ( Chair of the GB https://www.cncf.io/people/governing-board/ )
• You can talk to CNCF staff ( first row here - https://www.cncf.io/people/staff/ - Priyanka Chris Joanna Thabang )
• You can talk to Jim Zemlin ( https://www.linuxfoundation.org/about/leadership )
• You can talk to Nithya Ruff ( https://www.linuxfoundation.org/about/leadership - click on board of directors )

thanks, Dims

[mattfarina]

There are a few reasons I see the need to respond to this in order to provide clarification. (1) There is a proposal and it's fair to respond to it, (2) I think it's important to provide context for those who are reading this, and (3) people will find this in the future looking for TOC patterns to follow and I would like to be clear.

We are talking the equivalence of a set of individuals rebooting the TOC and starting a TOCv2 without the consensus of seating members

If we look at the listed maintainers of the Notary project when changes occurred or at any other time they are not the signatories on the letter above or those that have raised issues here.

If those who were previously maintainers would like to step forward and discuss the issues with us we would be happy to listen.

If someone has a case that maintainers were "rebooted" please contact us with details rather than general accusations. Notary has tracked their maintainers since prior to it being a CNCF project.

The concern in the quote above does not appear to be valid.

Due to the disconnect and failings in reparation, those members eventually gave up on recourse and started a new project outside of the CNCF which give birth to a separate foundation as they found the committee chartered with technical oversight wasn't willing to step in and exert technical oversight.

This is one description of the history that not everyone shares.

First, it's important to note the TOC's role. Projects are independently governed. It is not the place of the TOC to step into a project to direct how it should build or design something. When those in the community around a project have disagreements on how to proceed in design and development, it is up to the project maintainers to navigate the situation. The project maintainers are the ones empowered here.

The community was not in agreement around how to proceed - and we now have multiple projects that handle overlapping aspects while targeting use cases that are a little different and did things using a slightly different methodology. There is a rich history there that I'm not going to try to sum up in this comment.

It's important to note that if the other project(s) had been well suited for all the needed use cases, than the work in the notary project would not be active today for lack of a need for it. This isn't to diminish either project. It's to highlight that a diversity of use cases exist.

The other project could have come to the CNCF. A separate foundation, under the Linux Foundation, was a choice folks made. Competing projects in the CNCF already occur. The other project was not proposed to the CNCF.

I say these things to add context. I would be delighted if both projects were successful or if only one is. Problems being solved in ways that satisfy those with the problems is what I hope for. It's a market driven approach and there are some sub-markets there.

Said project has since superseded notation in traction, growth, and maturity.

This seems like a good time to note that the TOC does not crown kings. This is one of the TOC Principles. By not crowning kings the TOC enables innovation and a wide variety of use cases to be targeted. In the CNCF there are many competing projects today (e.g., containerd and CRI-O).

Since we are on the topic of the other project, it has not gone un-noticed that many of those related to the other project are the ones raising issues here. I would ask that those competing in this space be professional with each other. Even to the point of collaborating on shared libraries and specifications where it makes sense.

As such, I propose the following course of prompt action:

There are a couple concerns I have with the proposed actions:

1. They do not appear to take into account how the work on notary is being designed and proceeding. The separation between a specification and reference implementation where other implementations of the specification are being done. The suggestions appear to be lacking in context.
2. The actions also ask the TOC to step in and take the project in a different direction from the maintainers. The ask is for the TOC to not allow the project to be self governing. This would set a precidence for other CNCF projects as it would mean that projects are no longer self governing. And, to do all of this for something other projects have already done.

I understand there is a fair amount of passion around this topic. Using processes, precidents, and making decisions based around handling decisions in common ways helps the TOC to guide projects while enabling them to be self governing without passion around a single issue on a project causing the TOC to act inconsistently.

If I put my project maintainer hat on, I appreciate this because I know what to expect.

[JustinCappos]

If we look at the listed maintainers of the Notary project when changes occurred or at any other time they are not the signatories on the letter above or those that have raised issues here.

If someone has a case that maintainers were "rebooted" please contact us with details rather than general accusations. Notary has tracked their maintainers since prior to it being a CNCF project.

There are two nuances to your questions which I think are important to note.

First, none of the Notary project creators were members of either group. In fact, apart from Justin Cormack, no one who was actually a Notary project maintainer was still participating in the project at that point. He had just been elected to the TOC and as I understand it with the mass exodus at Docker, had taken on a heavier workload. So, Justin Cormack (the only "official" maintainer), effectively was absent throughout this entire process. In later versions of the maintainers Marina Moore, who is a signatory of the letter, is listed as a maintainer, but this is missing important historical context as is discussed in the next paragraph...

Second, and importantly, the TUF and Notary (v1) projects were very intimately related. Almost all KubeCon talks we gave to that point were joint between the two projects. In fact, if I counted correctly, I've given a joint talk on TUF / Notary V1 with five of the seven of the past Notary V1 maintainers. The Notary V1 project lived under the TUF github organization. We were obviously intimately involved with design and the project overall. We didn't give the Notary V1 folks maintainer roles in TUF or ask for them in Notary V1, because there was little need or point.

It's important to note that if the other project(s) had been well suited for all the needed use cases, than the work in the notary project would not be active today for lack of a need for it. This isn't to diminish either project. It's to highlight that a diversity of use cases exist. ...

I'd also like to say that at least from my standpoint, I'm happy to have the Notation project exist. I do think it should have the space to grow. I am also glad the TOC has stressed the importance of security review in a security focused project, which was one of the two major concerns I had.

The position of myself (and as I understand the other signatories) has never been for the TOC to do kingmaking. We do believe that the TOC should handle name conflict / misappropriation issues. Abdicating this sets a precedent seems to enable us to start a Notary V3 project, which just seems bad for everyone. I'd be quite happy to see the Notation project fail or succeed on its own merits.

[dims]

@JustinCappos thanks. our current guidance that @mattfarina laid out above [1] stands.

[1] https://github.com/cncf/toc/issues/981#issuecomment-1411128982

[anvega]

> If someone has a case that maintainers were "rebooted" please contact us with details rather than general accusations. Notary has tracked their maintainers since prior to it being a CNCF project.

The codeowners of the repositories for Notary v2 /notaryproject and /notation are not tracked by the master maintainter spreadsheet. When the changes occurred and for the longest time commit bits were restricted to two people. Still today the bus factor expected of an incubation project is not met at an organizational level.

There are a couple concerns I have with the proposed actions:

They do not appear to take into account how the work on notary is being designed and proceeding. The separation between a specification and reference implementation where other implementations of the specification are being done. The suggestions appear to be lacking in context. The actions also ask the TOC to step in and take the project in a different direction from the maintainers. The ask is for the TOC to not allow the project to be self governing. This would set a precidence for other CNCF projects as it would mean that projects are no longer self governing. And, to do all of this for something other projects have already done.

The proposed actions take into account that if the new spec currently under design and its eventual conformant implementations were to be evaluated against the criteria the TOC sets forth for incubation projects, it fails due diligence due based off how nascent it all is:

In production by at least three independent direct adopters which, in the TOC’s judgement, are of adequate quality and scope.

Docker Registry uses v1 but there is no documentation of even just one production adopter of v2. Where and who are they? No end user organization would have a production outage if the repo is migrated or mirrored to a new location.

Perception vs Reality: Is there lots of buzz, but the software is flaky/untested/unused? Does it have a bad reputation for some flaw that has already been addressed?

There is bad reputation for security deficiencies that haven't been addressed The project fails to report a process for reporting vulnerabilities on the project site There is no mechanism in place to support private vulnerability reports

A clear versioning scheme.

There is no versioning or release cadenance documented.

Specifications must have at least one public reference implementation.

You say this is in the works. Yes, The README of the project states the spec is not formalized or completed : "The Notary v2 project is in early development and design documents should not be considered final." Yes, one day it will undergo a security but right not there is simply not enough to audit against that would be defensible.

This seems like a good time to note that the TOC does not crown kings. This is one of the TOC Principles. By not crowning kings the TOC enables innovation and a wide variety of use cases to be targeted. In the CNCF there are many competing projects today (e.g., containerd and CRI-O).

In this instance you are giving a project unfair advantage and positioning it as the incubated Signing and Verification spec and reference implementation for CNCF without proper due diligence. You are crowning a king but not putting the project through the respective motions.

If I put my project maintainer hat of other CNCF projects, I feel its unfair and unjust that not all projects are held to the same set standards.

Again, if you really want to empower the Notation maintainers, I'm not asking to take the project in a different direction, but to give them a clean room and a name that is not associated with strife. They deserve that much.

Last, I keep hearing the word precedent. A precedent is something that sets a standard for future events. In the absence of something that hasn't been dealt one in the past, one we are advocating for an unprecedented situation to be reconsidered. To adhere by the doctrine stare decisis “no prior precedent” is not helpful when not enforcing the only documented guidelines for project maturity that exist. Failing to deal with something simply because it hasn't been dealt with in the past won’t get us far - in fact is signal of a tragedy of the commons. For what it’s worth, the stare decisis doctrine exists in case law rather than law based on statutes or regulations such as codes of principles and charters (on which binding precedential effects does not apply). If you were going for the former and not the latter, I invite you to think of this a case of appeal. Important to add, how Matt and Dims explain during the recorded meeting they personally like to go about making decisions actually neglect how the TOC has handled decisions in the past when rationale factoring morale and ethics was favored over precedent.

[dims]

Ok folks, i think we have reached the point of diminishing returns. Unless there is new information, let's wind down the discussions here.

If there are new folks or new information surfaces, we stay the course. 🙏🏾

@anvega : very sorry to hear you characterize the entire TOC the way you did. please take it up with other folks mentioned above already https://github.com/cncf/toc/issues/981#issuecomment-1426303732

[anvega]

We’ve had a good trajectory cultivating a positive cloud native community. That has involved holding others accountable even when it’s not about doing what feels most safe or comfortable, but about doing what is right.

I deeply care about the integrity of every member of the TOC and for the TOC to accomplish its goals. That is why it causes me distress to see an insular body (and the prestige associated with it) to without much thought push and rubberstamp projects with a +1 without caring elaborate a sentence or two as to why they lukewarmly endorse it, but will shy away from matters that require careful thought and consideration. There are several instances where the TOC has ignored recommendations from the TAGs in the push to incubate or graduate a project or preserve in those stages as it is doing here again now, carving out arbitrary exceptions and overruling on the fly.

As Chair of the TOC, the tone and the language by which you have expressed yourself in the exchanges here leaves a lot to be desired. You seem set on wanting to suppress discourse and muzzle those of us expressing dissent. I do take as your are the chair that you are speaking on behalf of the entire TOC but given the short interval within responses, I don’t get the sense the group has reconvened in the last two days to discuss the matter. I find it important to disabuse people from the convinced they taken and the sentiment that they are absolutely right about things. We’ve lost sight and appreciation for the nuance and the grey. Certainty takes the cake. Nobody is listening, clearly not you. Telling me you are sorry for how I feel is not just deflecting but is a classic example of gaslighting.

I may be coming across as severe in my judgement but I say it as I see it. Holding someone accountable hopefully gives them the gift of awareness. I’m really making an effort here to express that I care. What would be unfair from me is to take it up above you without giving you the chance to acknowledge it and own up to it.

If there are new folks or new information surfaces, we stay the course. 🙏🏾

I sincerely hope there is a typo in there. Why would anyone bother speaking up in an attempt to drive change if things will remain the same?

Two new elements of information:

It has been brought up to my attention that Notary hasn’t ever conduct an annual review which is required by the TOC for all incubated projects. Will you please request and schedule the project to conduct and present a review. Next’s month’s call seems like reasonable enough timeframe.

On rebooting maintainers, you can see where questions were raised asking how selection was made went unanswered. An explanation as to why the most actively involved contributors were excluded from the process. The request was also made for a more diverse group but that was also ignored. See: https://github.com/notaryproject/notary/pull/1609.

I'll leave you with two quotes to ponder:

“Sunlight is said to be the best of disinfectants.” - Supreme Court Justice Louis Brandeis, 1913

This mantra of transparency and good governance remains true today and is a hallmark of a strong civil communities. Naturally the issues I raise here are hard to confront. I understand the discomfort. But not because it's uncomfortable does it mean there are diminishing returns to discussing transparently.

"Moderation in the pursuit of justice is no virtue." Barry Goldwater, 1964

[dims]

thanks for your feedback @anvega. this has been very illuminating.

[mattfarina]

There is a lot of investment and care in this topic. I want to say that I appreciate that. I also cannot respond to everything said in the comments since the last time I commented. I've responded to some to try to add context. I think a shared context is important.

but will shy away from matters that require careful thought and consideration

I, for one, have put in a lot of time into research, talking with people, thinking through various situations, and considering outcomes and what the TOC is tasked with doing. I have spent more time on this than I anticipated. I talked with people, read conversations, dug through git and the history there, re-read TOC stances (precedents) and scope, and more.

I want to see this handled well with a focus on moving forward and any opinion I have I could write a very long justification on.

I ask that everyone appreciates how much research and effort the TOC has put into this. Even when people don't agree with the TOC on something. We have a wide variety of considerations to take into account.

Second, and importantly, the TUF and Notary (v1) projects were very intimately related. Almost all KubeCon talks we gave to that point were joint between the two projects. In fact, if I counted correctly, I've given a joint talk on TUF / Notary V1 with five of the seven of the past Notary V1 maintainers. The Notary V1 project lived under the TUF github organization.

TUF and Notary came into the CNCF as two different projects. It always seemed out of place for Notary codebase to live under the TUF organization as they were, from the CNCFs perspective, two different projects. I understand the close ties. But, that was an implementation detail rather than an organizational one. It's like two companies having a relationship with each other rather than one being a subsidiary of the other. There is no requirement for TUF and Notary to be tied together into the future. Any decisions around that are scoped to be made by the project maintainers.

Different groups in the CNCF have different roles and responsibilities. Projects are self governing with design decisions and how those change over time being left to the projects. Notary, as an independent project separate from TUF, operates within that model. This is why we opened issues for a governance remediation process and provided direction there. To aide projects that have issues making decisions that are part of their scope.

The codeowners of the repositories for Notary v2 /notaryproject and /notation are not tracked by the master maintainter spreadsheet.

If I look at the maintainers file that lists all maintainers across the sub-projects it appears to be reflected in what the CNCF holds. The notary governance documents a process for updating that list. I have asked that those who actually have access is resolved to the governance process if they are not aligned. This is something I will be checking in on.

It has been brought up to my attention that Notary hasn’t ever conduct an annual review which is required by the TOC for all incubated projects.

Sandbox projects get an annual review. Incubating and graduating projects do not get an annual review. Notary, being an incubating project, does not need to have an annual review.

If someone does not agree with this process around annual reviews I would ask them to raise it as a separate issue for discussion.

On rebooting maintainers, you can see where questions were raised asking how selection was made went unanswered.

I can see the questions are left unanswered on the issue. I also see that there were meetings that discussed this. When it comes to the separation of concerns, the maintainers are responsible for their governance and electing or otherwise changing their members. The linked issues shows traceability to their vote for a change. Whether they shared a response to those questions or not is not a justification for the TOC to step into a project and break it up against the wishes of the maintainers.

This mantra of transparency and good governance remains true today and is a hallmark of a strong civil communities.

Transparency is great. We are all imperfect at it. We can all improve on it.

One of the things I value in governance is one which is consistent and defines scope. We can then debate scope in general. For example, imagine a governance that said J walking was illegal except for Bob. He could J walk legally. That would be inconsistent and unfair governance.

In this case the TOC is treating notary in a consistent manner given the scope of the TOC, the scope of the projects, and how we interact with each other. Notary isn't being singled out and treated differently.

Numerous projects have created sub-project and iterated on their major version and design. All in the incubation phase. To single out notary for this would be inconsistent. When a sub-project or new version comes along it doesn't have any production users. It's new. Incubation projects have gone through this before. The TOC would be inconsistent to take one incident of this happening and treat it differently from other projects that have done it. That, in my opinion, would be unfair governance.

If someone thinks the TOC needs to change the way it handles things in ways that apply to all projects, please raise the issue with a proposal so that we can discuss it.

I appreciate the care people have for this topic. I also ask that people try to understand the underlying reasons why the TOC is making decisions the way it is. If we are unclear on the context, please ask more questions to derive the reasons. If you find a general issue in process, please raise it so that it can be discussed as it applies to all projects.

[anvega]

I appreciate the level of thought and depth of the response.

I’ll tell you why the optics are skewed as heavily one-sided. The two organizations holding control of the direction of Notary v2 at the time of the complaint were Microsoft and Docker. Those two organizations are represented in the TOC. Purdue, NYU, Chainguard, and Datadog are not. The complainants who felt they were cutout and requesting for involvement do not have any representation in the TOC. The only maintainer appointed rep who could have advocated their plight has a conflict of interest giving his participation in the project and his employer. Perceived lack of neutrality has been a constant undertone of this issue.

Given your proximity in the TOC to these individuals these were amongst the first people for you to talk to as well as more contributors or people representing the organizations that were choking control of the project. On the other hand, there are several of the complaints and collaborators to them who tell me they haven't spoken to you.

Does beg the question if whether either the composition of Notation maintainers or the TOC would be different, if the decisions stated above would remain the same.

TUF and Notary came into the CNCF as two different projects. It always seemed out of place for Notary codebase to live under the TUF organization as they were, from the CNCFs perspective, two different projects. I understand the close ties. But, that was an implementation detail rather than an organizational one. It's like two companies having a relationship with each other rather than one being a subsidiary of the other. There is no requirement for TUF and Notary to be tied together into the future. Any decisions around that are scoped to be made by the project maintainers.

It seems out of place for the Notation codebase to live in the same repository as Notary. There is no requirement for Notary and Notation to be tied together into the future. Or share a logo. If you are looking for precedent, Buildpacks had a security assessment for Incubation that included spec and implementation. Harbor and Falco had scrutinious reviews of all subprojects and components that encumbered the projects while at incubation. Then look at SPIFFE and SPIRE. The spec is one project and the reference implementation is another. It took a lot of back and forth and deliberation to fully decouple the two, but that is the case. One favorable outcome is having a Steering Committee at a spec level separate from the maintainers of the implementation. Another perk is having maintainer talk track slots at KubeCon for each.

Different groups in the CNCF have different roles and responsibilities. Projects are self governing with design decisions and how those change over time being left to the projects. Notary, as an independent project separate from TUF, operates within that model. This is why we opened issues for a governance remediation process and provided direction there. To aide projects that have issues making decisions that are part of their scope.

As one of the Technical Leaders for TAG Security, I'm expressly reiterating the use of anti patterns and plagued with security deficiencies in the project. There is a general consensus from the TAG leadership that Notation is currently flawed and not compliant with the security guarantees in line with an incubation project. Should the TAG be filling issues against the Notation issue tracker directly or should those still be proxied through the TOC?

I can see the questions are left unanswered on the issue. I also see that there were meetings that discussed this. When it comes to the separation of concerns, the maintainers are responsible for their governance and electing or otherwise changing their members. The linked issues shows traceability to their vote for a change. Whether they shared a response to those questions or not is not a justification for the TOC to step into a project and break it up against the wishes of the maintainers.

The projects are ultimately responsible for their governance but when at the time of the election and the time leading up to it discrimination occurs based on gender including the only woman in the ballot, by age; the youngest members of the community, by status; the only folks not compensated six figure salaries, for folks actively attempting to participate on the project and that were on the ballot for maintainership, it does seem a good reason for the TOC to step in. Not just a CoC concern.

No one is asking for Notation to be broken. Why the opposition to give it a logo and home of its own? Revisiting the proposal I made from the comment, if it's non-negotiable to respawn the project to sandbox, at least consider just the rename, a new logo, and a new website while still at incubation. Notation doesn't get set back but the people who feel they deserve reparation from the misuse of the name get to move on.

[mattfarina]

The two organizations holding control of the direction of Notary v2 at the time of the complaint were Microsoft and Docker. Those two organizations are represented in the TOC.

The TOC has been spending time discussing conflicts of interest. We want to better handle those situations. You can see an example of it in the work at https://github.com/cncf/toc/issues/1007.

In this case you've worked with Dims and I who don't have conflicts of interest in this situation. While I can't share all the communications of the TOC I can say that I have not seen any conflicts of interest arise. Those two individuals with conflicts of interest have left this situation to the rest of us.

The only maintainer appointed rep who could have advocated their plight has a conflict of interest giving his participation in the project and his employer.

I've previously been a chair of Kubernetes SIG Architecture and SIG Apps. I'm a maintainer of two CNCF projects and have been a maintainer for years. I've contributed to CNCF projects and been part of the community since before the CNCF existed.

While I'm not a maintainer elected rep I am a maintainer and can understand the situation and feel for all parties involved. I've maintained projects where we didn't go in a direction some members in the community wanted. I've also been frustrated when a project I contributed to went in a different direction than I thought it should have.

It's important to note that maintainer appointed rep is one who represents the maintainers. That means they represent those in the listed maintainers rather than those in the community of contributors. Who from Purdue, Chainguard, and so on was a maintainer of Notary?

Imagine a case where a group of contributors and those with opinions wanted to take SPIFFE or SPIRE in a direction the maintainers didn't want to. So the group went to the TOC. How should the TOC respond? Who should the TOC, and specifically the maintainer elected TOC rep, support? What if some of the people in that group were maintainers on other CNCF projects? If we look at the situation applied to other projects we can work out rules to follow and apply generally to everyone.

While we're on the topic of conflicts of interest, associations with other overlapping projects that compete in some spaces Notary does have been noticed here.

On the other hand, there are several of the complaints and collaborators to them who tell me they haven't spoken to you.

I have not personally spoken with everyone involved. Several members of the TOC have spoken with many who have complaints and we as a TOC talk with each other. I'm sure it's not everyone with a complaint. We, as a group, have spent considerable time listening to people involved.

Does beg the question if whether either the composition of Notation maintainers or the TOC would be different, if the decisions stated above would remain the same.

The TOC doesn't treat every situation as unique. We rely on common rules, processes, precedents, and scope. There is too much for the TOC to accomplish if we don't handle things in a common manner. Process and commonality are at the core for how the TOC functions. That is being applied here. How has the TOC treated situations in the past? What other projects have done similar things as Notary without the TOC having issue? How do scope lines fall between different parts of the CNCF? These are the kinds of questions I asked when looking into this situation.

If you look at the date on the original letter you'll see that it was addressed to a previous incarnation of the TOC, with many different members. They already looked into this Notary situation and responded. This is the second pass of looking at it. Two incarnations of the TOC have now addressed this.

The first time the TOC addressed this the details were not made public at the time. There was an intention to make it public but the actual work of doing so appears to have slipped through the cracks. For that, we apologize.

It seems out of place for the Notation codebase to live in the same repository as Notary. There is no requirement for Notary and Notation to be tied together into the future.

Whether a spec and reference implementation lives under one project or two is not something the TOC has decided a common pattern on. To come up with a common pattern would take considerable work and require guidance that makes it clear what the TOC means.

The TOC has left decisions on this separation to the project maintainers. We are not looking to change that, as a rule that applies to all projects, at this time.

As one of the Technical Leaders for TAG Security, I'm expressly reiterating the use of anti patterns and plagued with security deficiencies in the project. There is a general consensus from the TAG leadership that Notation is currently flawed and not compliant with the security guarantees in line with an incubation project. Should the TAG be filling issues against the Notation issue tracker directly or should those still be proxied through the TOC?

If the TAG found specific security issues with Istio or CRI-O (to pick two projects at random) would they contact the TOC to convey those issues and discuss them? I would expect the TAG would work with the project directly as it would with any other project.

Leaders in the CNCF should be trying to help projects succeed. When we work together it should be in a positive manner looking out for the best interest of the projects, their users, the maintainers, and the contributors. I would expect TAG Security to understand the scope of the Notary/Notation work, what the situations are to use it, and to help improve security there.

Are you saying that TAG Security leadership has taken the time to understand the context of Notary/Notation going forward, has tried to work with the maintainers to improve security where deficiencies have been found and clearly articulated, and been ignored or turned away by the Notary maintainers?

The projects are ultimately responsible for their governance but when at the time of the election and the time leading up to it discrimination occurs based on gender including the only woman in the ballot, by age;...

If you have found CoC violations please report them. Speculating on issues in public ranges from being non-actionable to FUD. No one has a way of telling and issues, if present, are left unaddressed. Please report any CoC violations.

If this isn't a CoC violation but something related to governance (since it's about elections) we are creating a process for those situations. Please directly report the details of any issue here. It would need to be thoroughly investigated as a start.

No one is asking for Notation to be broken. Why the opposition to give it a logo and home of its own?

There is a request from people who aren't maintainers of Notary (some of whom work on competing projects) to make changes to Notary in a way that goes against the wishes of the Notary maintainers. The ask goes against the TOC principle of projects being self governing.

If you have a request of the TOC please work within the principles and scope that already exist for the CNCF.

[sdake]

Hello fellow gang members,

Introduction

I want to thank the participants of this thread for their involvement and investment of time. Clearly there is passion in this thread. As an outside observer lacking specific facts, my perspective of this problem from someone that lives the principles and values of open source for a lifetime of engineering achievement (25 years of experience), and a lifetime Eagle Scout who lives the Boy Scout Oath:

There are a few issues with this thread:

• There are ad-hominem arguments.
• There is the appearance of misconduct by two very, very large engineering companies (Microsoft and Docker).
• There is an accusation of a conflict of interest.
• There is an accusation that the strong preyed on the weak.
• Some in this thread need to clarify the hat the respondent is wearing if it is unclear (individual contributor, TOC member, CNCF GB member, Linux Foundation member, Linux Foundation Staff, etc.).
• Finally, there is an accusation that big technology companies shut out academic researchers.

The misconduct in question appears to be:

1. Microsoft and Docker executed a hostile takeover of a project
2. After which, they redefined the project's charter, principles, values, mission, and staffing
3. After which, they fired the prior maintainers

There are additional complaints:

• The CNCF TOC is covering for errors in judgment that occurred in 2021 (2 years ago).
• The CNCF TOC asks the inflicted parties to disagree and commit.
• The CNCF TOC is focused on the future.

The corrective action asked for by those allegedly wronged:

1. The CNCF-owned trademark, Notary, should be discarded.

My response

I have sent a signed printed copy of this specific message to Jim Zemlin (j@zemlin) and Chris Aniszcyk (@caniszczyk) via US postal service, email, and github.

Without coloring the events with my own unfounded opinions (because I don't have the facts), I am requesting the Linux Foundation and its subsidiary, the Cloud Native Computing Foundation, to investigate the wrongdoing alleged in this thread and commit to a timeline for the following results:

• a transparent summary of the investigation results by an independent third party.
• If corrective action is needed, try to find common ground, such as an apology, with a professional or otherwise mediator rather than destroying value (the Notary trademark) to correct the problem.
• Ensure that this event can not and will never happen again.
• A written commitment from the Linux Foundation to execute an investigation into Girthub Issue 981 with a timeline and expected artifacts.

I write this because I find this problem's transparency to be lacking. Assuming for a moment the opinions on this issue are factual. If this is the case, the corrective action taken by the CNCF leadership is profoundly unjust and, therefore, undermines the integrity of the Linux Foundation, the CNCF, and the broader mission of open source.

Thank you, Steven Dake

[sdake]

I made an error in the message above, but I do not wish to edit it. Jim Zemlin's actual github id is:

@jzemlin

[caniszczyk]

On behalf of CNCF leadership, we would like to thank you all for the spirited debate here along with a lot of detailed opinions. In this matter, the @cncf/cncf-toc has published a proposed plan of action. Some of you are opposed to it and have a different set of goals you are aiming for. The CNCF leadership is open to hold a meeting to listen to everyone who would like to speak their case. Please come prepared with an alternative action plan that aligns with the CNCF principles, charter and documented tradeoffs.

However, the TOC are elected and TOC folks are always expected to do what is best for the cloud native ecosystem and CNCF. The TOC has responsibility in the end for "aligning projects, removing or archiving projects" per the charter and truly has the final say of all technical oversight in CNCF.

@amye will find space at an upcoming meeting or work to hold a special meeting for everyone to have a constructive dialog in a supportive atmosphere.

[sdake]

Hi @caniszczyk .

I don't have any standing in this argument besides being a long-time open-source leader.

The CNCF rejected a paper I wrote a long time ago. I asked for the record of why. I was so impressed by the speaker selection team's responses made of community members. Extremely thorough and very professional. And this was done for the thousands of papers submitted each KubeCon per paper, not to appease me. Very impressive.

The TOC members I know personally, many for most of my career, operate with moral absolutism and unquestioned integrity. This is generally the case across most open-source communities that have outstanding leadership.

The problem appears to be a split perspective of history. It is unclear to me why this occurred, and no party has published either version of the history. "look at git" is not the correct answer to this question.

With understanding the facts, taking the right action may solve the correct problem.

Instead of holding a town hall where various parties stump, can we get a written account of what went down?

Thank you, -steve

[anvega]

Thanks @caniszczyk. To wrap up here for now, here is the response to the rebuttal of my previous comment.

If you have a request of the TOC please work within the principles and scope that already exist for the CNCF.

That’s why this issue was opened up in first place and what has been debated throughout. It can quickly become recursive to open separate issues to be told to open yet more separate issues.

If this isn't a CoC violation but something related to governance (since it's about elections) we are creating a process for those situations. Please directly report the details of any issue here. It would need to be thoroughly investigated as a start.

We directly reported detailed observations on irregularities on how the Notary project is governed. The stance from the TOC seems firm that this has been addressed twice already and won't look into it further.

Can you share a draft of that process for those situations that is being created? It's positive to hear signaling that the work is being done, but would be even better signal to see the work. Everyone would agree it would be beneficial for the authoring of said process to be done in public with community input.

If you have found CoC violations please report them. Speculating on issues in public ranges from being non-actionable to FUD. No one has a way of telling and issues, if present, are left unaddressed. Please report any CoC violations.

Several of us participating on this threat including you and I are involved in the CoC workgroup. “If you see (or hear) something, say something”. I won’t ask whether either one has acted upon that as mention or reference to any ongoing investigation could compromise the integrity of the process.

I have not personally spoken with everyone involved. Several members of the TOC have spoken with many who have complaints and we as a TOC talk with each other. I'm sure it's not everyone with a complaint. We, as a group, have spent considerable time listening to people involved.

I heard from one of the people interviewed that the interviewer expressed no interest to speak to one specific complaint and I know for a fact of two others who were not sought out. I'm still perplexed by how can the TOC form a complete picture without hearing from everyone involved.

If you look at the date on the original letter you'll see that it was addressed to a previous incarnation of the TOC, with many different members. They already looked into this Notary situation and responded. This is the second pass of looking at it. Two incarnations of the TOC have now addressed this.

There are many of the same members in both configurations of the TOC then and now.

If the TAG found specific security issues with Istio or CRI-O (to pick two projects at random) would they contact the TOC to convey those issues and discuss them? I would expect the TAG would work with the project directly as it would with any other project.

Leaders in the CNCF should be trying to help projects succeed. When we work together it should be in a positive manner looking out for the best interest of the projects, their users, the maintainers, and the contributors. I would expect TAG Security to understand the scope of the Notary/Notation work, what the situations are to use it, and to help improve security there.

Are you saying that TAG Security leadership has taken the time to understand the context of Notary/Notation going forward, has tried to work with the maintainers to improve security where deficiencies have been found and clearly articulated, and been ignored or turned away by the Notary maintainers?

The entirety of the TAG is intimately familiar with the use cases and sharp edges of Notary. The project was closely evaluated at the time of writing the Supply Chain Best Practices White Paper and the Secure Software Factory Reference Architecture. It was found that due to its shortcomings it could not be postulated as a fit or reference.

Both Justin and Marina who are Technical Leaders in the TAG have repeatedly attempting to work with the maintainers. You will hear the same from Michael Liebermann and Brandon Lum who have tracked matters closely all long. It wasn't just Justin's and Marina's attempts to work with the maintainers that were disregarded, but code contributions were entirely ignored and never received a review - those pull requests were eventual closed and redirected to /go-tuf.

If the TAG found specific security issues with Istio or CRI-O (to pick two projects at random) would they contact the TOC to convey those issues and discuss them? I would expect the TAG would work with the project directly as it would with any other project.

That impetus for Justin opening this issue stems from TAG members attempting to work directly with the project but being ignored and turned away. That is voiced as him representing the TAG as well as the plight of students in his PhD program.

If the TAG were to find issues with any other projects, we'd take the same approach of first getting to know the project team, attending their calls, developing shared context, and then bring up the issues. We may lean on the TOC for a warm intro to not come in abrasively as unknown outsiders.

Imagine a case where a group of contributors and those with opinions wanted to take SPIFFE or SPIRE in a direction the maintainers didn't want to. So the group went to the TOC. How should the TOC respond? Who should the TOC, and specifically the maintainer elected TOC rep, support? What if some of the people in that group were maintainers on other CNCF projects? If we look at the situation applied to other projects we can work out rules to follow and apply generally to everyone.

Exactly what we are talking about here. TUF and Notary were started alongside each other as a joint endeavor (see link). TUF as the spec and Notary as the implementation. With the sequestration of Notary and the spawn of Notary v2, the two projects suddenly were out of alignment. If SPIRE were to perform mitosis without respect the SPIFFE specification and call it SPIRE v2 that would be counterproductive to all involved. If someone wanted to start a new spec with its own reference implementation they could do that well under a new project.

It's important to note that maintainer appointed rep is one who represents the maintainers. That means they represent those in the listed maintainers rather than those in the community of contributors. Who from Purdue, Chainguard, and so on was a maintainer of Notary?

You can check the maintainers.csv and see the list of maintainers of TUF for which Notary alleges to conform with. Regardless of that, you said the maintainer rep has left the situation to the rest of you, which means there is not a maintainer rep for maintainers as a whole.

The TOC has been spending time discussing conflicts of interest. We want to better handle those situations. You can see an example of it in the work at #1007.

The community also wants the TOC to have a better handle on those issues, hence why I prompted the discussion in first place. I wish this discussion had been proactive and I didn't have to flag it to see a reaction. Although I praise how receptive the TOC has been. I'm encouraged to hear time has been carved out to dedicate focus to the topic in what’s been 5 days since last Thursday.

[jkjell]

I have been intently watching this issue over the last fews months, and when the past issue was filed. I have been afraid to say anything due to the sensitive nature and large audience of this conversation. I know that my knowledge of this situation is incomplete but, I am confident enough in what I know to make the below statement. I also feel obligated to state that my comments below are my own and not the views of my employer, VMware.

The main problem with these issues is that we have been, and continue to be talking about something that has spanned projects and even foundations. This is not a Notary v2 issues. This is also not solely a technical issue. Most of what has transpired over the past two years has never been publicly documented or disclosed. We are continually talking about this tacitly, leaving casual observers to wonder what is really going on.

This puts folks on all sides in an untenable situation. Any resolution coming from the TOC will be unsatisfactory to the aggrieved because it, by design, has no room for empathy towards what they suffered. Any resolution coming from past "precedent" will be unsatisfactory because what has happened was unprecedented.

Damage has been done. People have been emotionally harmed, motivations to continue contributing has been stymied, and the community has suffered. Now, because of the narrow lens of this examination, we're continuing to do damage.

I think it's time to call out that there's a larger problem here. Acknowledge that it's going to require a holistic examination and a unique solution. Let's remove the self imposed boundaries of our projects, committees, companies, and foundations to come together and solve this problem. It will make us all better contributors, community members, and most importantly, human beings.

[lumjjb]

If the TAG found specific security issues with Istio or CRI-O (to pick two projects at random) would they contact the TOC to convey those issues and discuss them? I would expect the TAG would work with the project directly as it would with any other project.

Towards finding a solution, I want to make sure to untangle issues of security and governance, and how TAG Security is involved with them pertaining to projects. The TAG chimes in and highlights security issues and concerns, some of these concerns have some overlap with governance (e.g. unilateral write access, etc.). Given that there is an ambiguous line between the scope of concerns around security vs governance, it will be helpful to further define this. When it comes to governance, it is not something that the TAG has the positioning or the capabilities to assist in, the same as how we would not have the ability to help with a storage problem.