Support Tekton Pipelines as a backend

[terrytangyuan]

This issue is for discussing adding Tekton Pipelines as a backend in Couler that's consistent with the design proposed in https://github.com/couler-proj/couler/pull/24.

If you are interested in contributing, please comment and share your proposals below. Once we agree on a concrete implementation plan, we can then split this into multiple issues so more people can get involved.

[chinazj]

hi terrytangyuan

This are all fields of taskRun

  - `serviceAccountName`
  - `params`
  - `resources` 
    - `inputs` 
    - `outputs` 
  - `timeout` 
  - `podTemplate` 
  - `workspaces`

It will be presented like this in the couler

def step()
     tekton.run.SetServiceAccountName()
     tekton.run.SetParams()
     tekton.run.AddResources.inputs()
     ...

[chinazj]

Task in tekton contains multiple steps. How does this express in couler?

This is what i think，addConmand represents a step in tekton's task

def step()
    tekton.run.addConmand("echo hello", image) 
    tekton.run.addConmand("docker pull",image)

[terrytangyuan]

@chinazj We had similar discussions in https://github.com/couler-proj/couler/pull/24#discussion_r481227856? Does that help?

Also cc @FogDong @xinbinhuang for any suggestions/feedback/proposals.

[FogDong]

In the latest v0.16 version of Tekton, the condition field is deprecated and we use whenExpression instead.

The components of WhenExpressions are Input, Operator and Values:

• Input is the input for the WhenExpression which can be static inputs or variables (Parameters or Results). If the Input is not provided, it defaults to an empty string.
• Operator represents an Input's relationship to a set of Values. A valid Operator must be provided, which can be either in or notin.
• Values is an array of string values. The Values array must be provided and be non-empty. It can contain static values or variables (Parameters or Results).

It's quite different from our when(cond, if_op, else_op) design.

There is another question I want to talk about, the DAG in tekton and argo. In tekton, we use runAfter in Pipeline to control the DAG, should we have a consistent function and then convert it to a different method on the back end?

[chinazj]

@chinazj We had similar discussions in #24 (comment)? Does that help?

Also cc @FogDong @xinbinhuang for any suggestions/feedback/proposals.

yes.Following their ideas, I just made concrete

[terrytangyuan]

@FogDong Regarding when, perhaps we can start with implementing when(cond, if_op) without the else_op where cond is essentially the conditional expression using components of WhenExpressions, e.g. $(params.path) in README.md, and if_op is the actual task that will be executed when the condition is met.

For DAG, yes I agree that we can start off by writing something that works well with Tekton and then wrap it in a consistent interface for various backends (it's fine to have some feature parity at the beginning since we are still prototyping).