search

tektoncd / pipeline

A cloud-native Pipeline resource.
https://tekton.dev
Apache License 2.0
8.38k stars 1.77k forks source link

Task trigger within a pipeline #1136

Open afrittoli opened 5 years ago

afrittoli commented 5 years ago

This is a proposed feature that would help orchestrating asynchronous pipelines.

The use case is as follows: as a user I want to be able to include in a pipeline a component that is executed asynchronously and possibly externally i.e. Tekton on a different cluster or something that is not Tekton at all.

For instance a pipeline may include a Task that sends a cloud event on completion. The cloud event triggers an external system that does a certain amount of processing, and sends a call back via cloud event. In the pipeline below, tasks A and C are started in parallel. Task B is only executed once A is complete. Task D is only executed when a cloud event is received. In this specific example this event is a call-back from an event that was sent by Task C to an external system.

task_trigger_in_pipeline

In this example the pipeline helps us orchestrate the execution of an external task, since the overall pipeline condition exposed for the pipeline resource via the k8s API includes the execution and result of a process on an external system.

The UX model I had in mind for this feature is that a CloudEventPipelineResource could be extended to be used as an input, include a reference to a template, and Tekton would setup a binding implicitly for the Task that as that input resource.

I submit the same to the triggers project since I think the implementation would be across the two projects. https://github.com/tektoncd/triggers/issues/44

vdemeester commented 5 years ago

@afrittoli I wonder if this makes Pipeline (and resources) too complicated. If we have triggers that consume CloudEvents, the Task D could be it's own pipeline, independent of the rest of the pipeline, and know nothing on how to consume CloudEvents.

EliZucker commented 5 years ago

Also thinking something similar to @vdemeester . The idea for the trigger project right now is allow for a general way to accept events like CloudEvents, then use specific fields from the event payloads as parameters when invoking external resources. To me it seems harder to allow for a raw CloudEvent payload to enter a task as an input.

afrittoli commented 4 years ago

@afrittoli I wonder if this makes Pipeline (and resources) too complicated. If we have triggers that consume CloudEvents, the Task D could be it's own pipeline, independent of the rest of the pipeline, and know nothing on how to consume CloudEvents.

Heh, that's exactly the point. Task D could of course be an independent TaskRun or PipelineRun, however it's success status would be then decoupled from that of the initial pipeline, so some extra component would be required to decide whether the overall pipeline was successful or not.

afrittoli commented 4 years ago

Also thinking something similar to @vdemeester . The idea for the trigger project right now is allow for a general way to accept events like CloudEvents, then use specific fields from the event payloads as parameters when invoking external resources. To me it seems harder to allow for a raw CloudEvent payload to enter a task as an input.

Yeah - I didn't intend for a TaskRun to implement a new way of accepting CloudEvents - this is merely a way to say that a TaskRun may depend on an external input. The external input might be a well defined interface between tektoncd/triggers and tektoncd/pipeline. In fact I was thinking that we could have the taskrun controller creating an event listener dedicated to feeding the input to the taskrun.

vincent-pli commented 4 years ago

@afrittoli

fact I was thinking that we could have the taskrun controller creating an event listener dedicated to feeding the input to the taskrun.

I think it's make sense. we can have a CloudEvent input resource (actually a listener). then we could have an event receive Condition, will hang if no right event received. Take these two thing together, we can achieve the upper diagram.

vdemeester commented 4 years ago

@afrittoli what is the status of this issue. It is technically possible with our current components (tektoncd/pipeline and tektoncd/triggers).

afrittoli commented 4 years ago

While it might be technically possible, but it would be a bit hacky and inefficient.

I thought it would be a nice feature to have, as it would be one way of pausing a branch of a pipeline on a potentially delayed input. I'm happy to keep this in the backlog - there may be more interest about it in future.

tekton-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

afrittoli commented 3 years ago

Custom tasks may come in handy here.

afrittoli commented 3 years ago

Going back to the idea of using a custom task. If the remote service supports webhooks (like GitHub or GitLab), this could be solved using a custom task controller that manages triggers. The custom controller would: