Proposal: Cloud Events Controller and CDEvents

afrittoli commented 3 years ago

Feature request

Introduce a controller dedicated to sending CloudEvents in the format specified by the CDF SIG Events / CDEvents, which can be later extended to support multiple formats of CloudEvents and events specific options and features.

Tekton defines low level abstractions like Tasks and Pipelines. The SIG Events protocol defines higher level abstractions too, like “Change”, “Build”, “Deployment” and more. When Tekton is used through an opinionated CI/CD system on top, Tekton would send the events related to tasks and pipelines only. When Tekton is used standalone, we would like to provide users a mechanism to identify their Tekton Tasks and Pipelines as implementations of higher level abstractions, and have Tekton send the corresponding events too. This would be most likely implemented through some form of annotation on Tekton resources.

We would like to start with a project in the tektoncd/experimental repo, with the end goal to either incorporated into tektoncd/pipeline or alternatively become a new project in the tektoncd org. Initial owners of the project will be Vibhav Bobade and Andrea Frittoli.

As described in the community repo, the first step is to present the idea in a WG. This was done in the WG on Wed May 26th. The next step is to create an issue in the community repo.

Use case

The CDF SIG Events mission is to define a common event based protocol for CI/CD systems to interoperate. An initial draft of the protocol specification is available on the SIG repo. The protocol is based on CloudEvents.

Tekton can already generate CloudEvents for several events, but the format of such events is Tekton specific, and cannot be customized.

Interoperability is key to the success of Tekton, and supporting this new format of CloudEvents would help DevOps engineers and architects integrating Tekton with other CI/CD systems as well as collecting metrics from heterogeneous CI/CD workflows.

Additional Information

It was already proposed to move Tekton support for CloudEvents to a dedicated binary / controller. This work could align with that proposal and implement a controller that could send both the existing format as well as new formats of CloudEvents.

afrittoli commented 3 years ago

/cc @waveywaves @imjasonh @salaboy @bobcatfish

waveywaves commented 3 years ago

LGTM !

vdemeester commented 3 years ago

LGTM 👍🏼

bobcatfish commented 3 years ago

Sounds good to me! 👍

Curious about a couple things:

Would you see this controller eventually doing all the cloud event emitting? (I'm wondering if going this route, it might make sense to remove the cloud event emitting functionality from the existing pipeline controller, and make it the job of this controller only? Could be a bit cleaner?)
I'm not totally clear on what it will look like for this controller to emit some other 'higher level abstraction based' events based on annotations, curious to hear more, but maybe that's a WIP (it sounds a lot like this might be about some kind of traceability? i wonder if that's a problem to tackle on its own? i.e. knowing what triggered what all the way from start to end)

afrittoli commented 3 years ago

Sounds good to me! 👍

Curious about a couple things:

Would you see this controller eventually doing all the cloud event emitting? (I'm wondering if going this route, it might make sense to remove the cloud event emitting functionality from the existing pipeline controller, and make it the job of this controller only? Could be a bit cleaner?)

Definitely yes. I would expect eventually to move all cloud-events related logic to a dedicated controller and remove it from the main controller. I would like eventually to have this controller optionally perform extra logic on events, such as store them in tekton results and more.

I'm not totally clear on what it will look like for this controller to emit some other 'higher level abstraction based' events based on annotations, curious to hear more, but maybe that's a WIP (it sounds a lot like this might be about some kind of traceability? i wonder if that's a problem to tackle on its own? i.e. knowing what triggered what all the way from start to end)

The general idea I have is that tasks / pipelines could be annotated (already in the catalog?) as responsible for a certain activity defined by the protocol (for instance one of https://github.com/cdfoundation/sig-events/blob/main/vocabulary-draft/continuous-integration-pipeline-events.md). The cloud events controller would detect such annotation and emit more events accordingly, for instance:

pipeline started
pipeline running
task started
build queued
build started
build finished
task finished
task started
test started
test finished
task finished
artifact published
pipeline finished

bobcatfish commented 3 years ago

Definitely yes. I would expect eventually to move all cloud-events related logic to a dedicated controller and remove it from the main controller.

that sounds amaaaaaazing 💯 👍 👍 👍

thanks for explaining re the annotations too, makes sense to me, esp. that its based on the a protocol being defined outside of tekton. (I wonder if eventually we might want to have a way of indicating that tasks are of these types, e.g. a "build" type or "test" type but a) that's probably a different story and/or b) maybe the annotation is all we need for that)

anyway sounds great!!

dibyom commented 3 years ago

LGTM!

afrittoli commented 3 years ago

Great, thanks for the feedback!! The initial PR is https://github.com/tektoncd/experimental/pull/754.

tekton-robot commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

tekton-robot commented 2 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

tekton-robot commented 2 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

tekton-robot commented 2 years ago

@tekton-robot: Closing this issue.

In response to [this](https://github.com/tektoncd/community/issues/435#issuecomment-993369957): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen` with a justification. >Mark the issue as fresh with `/remove-lifecycle rotten` with a justification. >If this issue should be exempted, mark the issue as frozen with `/lifecycle frozen` with a justification. > >/close > >Send feedback to [tektoncd/plumbing](https://github.com/tektoncd/plumbing). Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

afrittoli commented 2 years ago

/remove-lifecycle rotten

afrittoli commented 2 years ago

/lifecycle frozen

afrittoli commented 2 years ago

/reopen

tekton-robot commented 2 years ago

@afrittoli: Reopened this issue.

In response to [this](https://github.com/tektoncd/community/issues/435#issuecomment-1022233252): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

afrittoli commented 2 years ago

In order for this to address https://github.com/tektoncd/pipeline/issues/2944, I would l think we shall include support for Tekton cloudevents as well. Proposed next steps would then be:

reach functional parity between the experimental controller and the inbuilt one
support cdevents and tekton cloudevents through configuration
move the controller into tektoncd/pipeline - add a flag (default to false) about using the controller instead of the inbuilt one

The last step is a bit controversial: while it makes it easy to deploy the new controller, it moves it under the ownership of the tektoncd/pipeline team, which the team may or may not want to accept. It also may slow down experimentation with new features in the controller. Alternatives could be to move the controller to a dedicated repo, and eventually integrate it in the controller. The downside of this approach is that users would have to install an extra component and configure it independently of tektoncd/pipeline.

/cc @waveywaves

afrittoli commented 1 year ago

tektoncd / community