search

cncf / toc

⚖️ The CNCF Technical Oversight Committee (TOC) is the technical governing body of the CNCF Foundation.
https://cncf.io
1.66k stars 629 forks source link

Add proposal to archive Notary project #272

Closed charlesrichard closed 3 years ago

charlesrichard commented 5 years ago

The Notary project, currently in Incubating status, has seen a significant decline in contributions, including active maintainers, since it was added to CNCF in 2017. This decline drops below the bar set for Incubating projects including "Have a healthy number of committers" and "Demonstrate a substantial ongoing flow of commits and merged contributions." While both of these attributes are subjective, it is clear by comparing Notary's DevStats Health Table, its metrics more closely match a soon to be archived projet rkt, rather than the vibrant ecosystem of healthy projects including CRI-O, TUF, Helm, Harbor, and others. Given the aforementioned, including the last Notary release date being over a year ago and the lack of interest in Notary from its corporate founder, Docker Inc, I propose we archive the Notary project. I offer an alternative and other musings below.

A member of the community, Justin Cormack, publically stated that Notary needs more active maintainers and that a reason for the lack of contributions is that Notary is a "relatively mature project." One may read this as Notary is "done." If this is the case, rather than archival, one could reasonably expect a "mature" project to be a candidate for CNCF Graduated status under the CNCF process guidelines re: "stable" projects. Since it is not the intent of CNCF to incubate projects indefinitely, it's reasonable to require the Notary community to do what's necessary to graduate rather than getting by in what is an OSS purgatory.

Why is this important? The CNCF has a unique role in the community in helping organizations select the right tools to get the job done, very much represented by the CNCF Cloud Native Landscape and the CNCF hosted/endorsed projects. This representation implies a level of quality and safety to those wishing to adopt those technologies. In a lot of ways, CNCF is a rubber stamp of approval. As one who now works for IBM and the moniker that comes with it, I'll extend it to CNCF-> "No one gets fired for choosing a CNCF project." Coming full circle, with the unique role CNCF plays in the community comes a unique responsibility in that the projects CNCF endorses must be of a certain standard, which is why it is crucial for these projects to remain healthy on all facets. As such, indefinitely incubating in OSS purgatory only because a project once qualified to a certain standard but no longer does is not acceptable, breaking the credibility and goodwill of the role CNCF provides to the community.

lizrice commented 5 years ago

Notary is overdue a review anyway so TOC will schedule that review.

Until the TOC does that review, it would be appreciated if people could stop reaching the very premature conclusion that we're going to accept this proposal.

Comments from community members who have experience with Notary are very welcome.

justincormack commented 5 years ago

As a maintainer of the Notary project and an employee of Docker, I would like to directly address the implications in the above proposal; that somehow Notary is not a healthy OSS project. One should most importantly judge an open source project health by its usage and usage beyond the company that might have originally sponsored it. Docker, Microsoft Azure and IBM are a set of companies that use Docker in their products and Harbor, itself a healthy CNCF project, uses it as a component and extension any company using Harbor e.g. VMware/Pivotal and beyond are using Notary.

Specific for Docker about our usage of Notary it serves as an essential component in Docker Hub and in our Docker Enterprise offering. Notary is used to sign our official images and all certified/verified content (from the likes of IBM, Oracle, Microsoft, SAP etc.) hosted in Docker Hub which are pulled in aggregate over a 1.5 Billion times a month by over our millions of users of Docker Hub. Having addressed the clear health of Notary; I want to address contribution and how we would like to see that move forward.

There is ongoing active discussion within the maintainers and the companies that use Notary in their products around changing the design to integrate more closely into the OCI formats rather than having Notary as a standalone registry service. This is a big architectural change, and requires some changes in the TUF specification, and there is ongoing discussion and planning around this work with interested parties. This would definitely make integration into Kubernetes easier, which many people have asked for. There is significant interest in Notary, and people are starting to understand that supply chain risk is a significant and increasing issue in cloud native deployments, so interest in tooling for this is definitely increasing. We definitely want to move to graduation, but we currently think it would make sense to work on some of the architectural changes first.

JustinCappos commented 5 years ago

I'd like to second Justin Cormack's thoughts here.

From the TUF side, we're very conservative about changing a widely used specification that does its job well. So, while we deliberate quite a lot on changes to the spec with different adopters, we are conservative about the actual changes we ask from our adopters, such as Notary. Notary is very widely used in the cloud native space, accounting for most of our use in this domain and "moving too fast" for non-security critical items would be a potential problem for both TUF and Notary. I would be worried if there were a lot of commits / feature adds for such a project.

We've been actively working with the Notary team to ensure features are needed and added only in a way that does not compromise the security of Notary / TUF. For example, part of the more recent changes involve deeper integration with supply chain security products like in-toto ( https://in-toto.io ). There is an on-going discussion on this topic, led by Trishank Kuppusamy at Datadog ( https://www.datadoghq.com/blog/engineering/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto/ ), which we are working to standardize across the projects ( https://github.com/in-toto/ITE/pull/4 ). We are adding these changes in a way that we are sure we can fully reason about the implications before we ask our adopters both inside and outside the cloud native space to adopt them as well.

From where I'm sitting, Notary is a widely used project that serves its purpose well and has provided helpful protection when an attack could have otherwise had devastating impact. In my book, that sounds like a healthy project.

SantiagoTorres commented 5 years ago

I second what @justincormack and @JustinCappos said. Sharing the bar for security projects and otherwise is not ideal, and github-generation metrics (stars, commits, forks) will always put systems security projects in a bad light.

"move fast and break things" is the opposite of what you want with Notary.

technosophos commented 5 years ago

I am opposed to archiving a stable and highly useful project. It sends the signal that the value of a piece of software is how much it changes, not whether it is solving the problem it set out to solve. We've been happily using Notary, and feel that it is already a stable and reliable part of our toolchain. I don't see why archiving a production-grade project with plenty of usage is a good idea.

As anecdotal evidence, I once wrote an HTML5 parser that involved a huge amount of up-front work. After reaching standards compliance, we (the authors) found very little to work on. But the project wasn't dead by any means. Now, several years later, it still gets hundreds of thousands of downloads while we might merge only one or two PRs a year. Construing that as a failure would be absurd.

mattfarina commented 5 years ago

Something that is overlooked is that Notary is an incubating project. Could there be a push to have it as a graduated project? A graduated project is very much process based and is something I think notary can go through.

Aside from graduation, you chase after what you measure. It ends up setting goals and expectations. It's been almost 5 months since grep, the GNU version most Linux users use, has seen a commit. Would anyone think to archive it? When software is stable and meets a need it often doesn't need regular changes.

The CNCF project health dashboard prioritizes activity and growth. That leads us to mentally think this needs to happen for healthy projects. This is often not the case for mature maintained stable software. Not everyone and every projects wants to continually rewrite the things. I wonder, how should the CNCF address cases like this?

lumjjb commented 5 years ago

We (IBM Research) are still making new innovations on top of the notary as a foundational component. However, many of the things that we do on top of notary do not require any changes in notary itself since it is targeted security component that does its very specific functionality well. In that sense, we don't see there to be a need in any massive feature changes. Like most crypto related components, the regularity of standard changes are not often - which often go under a ton of scrutiny before being agreed upon.

I think the maintenance issue raised for notary seems more like a temporary short-term issue which could be addressed given the amount of usage in the community.

raravena80 commented 5 years ago

👎 for archiving 👍 for keeping it

The release cycle of security components tends to be slower imo, which maybe lead to the misinterpretation that the project is not useful.

danmx commented 5 years ago

Even though the project has not a lot of contributions it still can continuously maintain its quality. For example:

danmx commented 5 years ago

Some platforms even offer fuzzing for free for open source projects: https://fuzzit.dev/2019/08/12/announcing-rewards-for-go-rust-oss-projects/

SantiagoTorres commented 5 years ago

I agree that fuzzing is a good idea. I've been meaning to submit TUF and in-toto to oss-fuzz (and I may borrow Envoy's idea of making it a gsoc project :)).

I think most projects within the CNCF are not being actively fuzzed either.

trishankatdatadog commented 4 years ago

I am against archiving Notary, too. In fact, Notary needs to fully support all of the features of the TUF specification in order to accommodate future needs like supply chain security. As such, we expect to make and see changes in Notary relatively soon.

amatestuser123 commented 4 years ago

Reading the comments here, I think there might be some additional things that should be considered here. To me, Notary needs some short term additional help with maintenance to be considered an active project, in the longer term.

Active projects focused on security, should get a level of maintenance required to address outstanding vulnerabilities at a minimum.

Some specifics:-

Whilst these aren't necessarily serious things, they're still missing CVEs in a security project.

More generally on activity

There are responses to this issue from large corporations (Docker, IBM). Hopefully, they would be able to commit to providing resources to maintain the project.

squillace commented 4 years ago

https://github.com/cncf/toc/issues/272#issuecomment-519103032.

amatestuser124 commented 4 years ago

Hi All,

Is there any update on the status of Notary? This issue is almost a year old now, and looking at the Notary there have been no releases in the previous 12 months, meaning that issues noted above are still present and additionally there may be more issues from underlying libraries which have emerged in the last year.

squillace commented 4 years ago

The next version of notary is in process. @justincormack

trishankatdatadog commented 4 years ago

The next version of notary is in process. @justincormack

Yes, please follow us on GitHub and CNCF Slack

amatestuser124 commented 4 years ago

@squillace @trishankatdatadog thanks for that very interesting. Looking at that, would it be right to say this is an entirely new version that's currently being designed, so there's no plan for more released in Noary v1?

trishankatdatadog commented 4 years ago

@squillace @trishankatdatadog thanks for that very interesting. Looking at that, would it be right to say this is an entirely new version that's currently being designed, so there's no plan for more released in Noary v1?

Best to ask @justincormack. It's still early days for Notary-v2.

justincormack commented 4 years ago

Hi,

There will be another release for Notary v1 shortly. Notary v1 is used in production and in general we are not looking at adding new features as that work is going into the v2 process, which there is a lot of work going on in at the minute.

Justin

On Tue, Jul 28, 2020 at 9:53 PM Trishank Karthik Kuppusamy < notifications@github.com> wrote:

@squillace https://github.com/squillace @trishankatdatadog https://github.com/trishankatdatadog thanks for that very interesting. Looking at that, would it be right to say this is an entirely new version that's currently being designed, so there's no plan for more released in Noary v1?

Best to ask @justincormack https://github.com/justincormack. It's still early days for Notary-v2.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cncf/toc/issues/272#issuecomment-665277068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADVYPGJG3A3AI6GOQBC2JLR543FTANCNFSM4IJB4KYA .

squillace commented 4 years ago

I was unaware that only versions could be in the cncf as opposed to projects. Though I do appreciate your persistence, @amatestuser124, I reject every single premise on which your points have been based so far. I don't run the show, however, so it is merely my own opinion.

amatestuser124 commented 4 years ago

@squillace I think you think I'm trying to make a point that I'm not, so I'm not really sure what you're rejecting?

My only interest here is trying to establish where the Notary project is going, and whether the existing released project is still active and should be used/deployed/recommended to companies, as that point is unclear to me.

This issue seemed like the best place to raise this point as Notary is a CNCF project and this was started to discuss Notary's direction.

squillace commented 4 years ago

Fair enough! Notary v1 is the basis behind almost every major repositories signing stories -- so it makes sense to ask. The context of the OP, however, was to my mind noxious from the beginning, so I apologize if I have cast aspersions I shouldn't have.

caniszczyk commented 3 years ago

closing due to inactivity