kabanero-io / kabanero-foundation

This repo will be archived soon.
4 stars 19 forks source link

proposal: pipeline as a first-class kabanero citizen #60

Open gireeshpunathil opened 5 years ago

gireeshpunathil commented 5 years ago

At present, the pipeline is an instance of tekton pipelines customized for kabanero projects. Few pain points with this approach:

The proposal is to integrate pipeline as a first-class citizen into kabanero capability by providing a high level abstraction on its ingredients: more specifically, the tasks and resources. The objective is to expose an interface through which a user can define their own pipeline with the help of interactive with an intuitive workflow model.

Proposal 1:

Define a custom grammer, specification and a template for defining pipelines that the user can customize. It still sues YAML syntax, but the verbs and qualifiers are made more simpler, and high level.

Expose a command line interface that will compile the high level pipeline description manifest, and generate a fully-blown working set of pipelines that can be applied to k8s clusters.

Proposal 2:

Expose a command line interface that interacts with the user and constructs a pipeline from scratch, through a series of questions. Much more flexible than first one, but much more complex to implement.

these can be a fit for upstream tekton/pipeline project as well, but the advantage of kabanero are:

LMK WDYT!

stephenkinder commented 5 years ago

@gireeshpunathil I'm wondering if perhaps there is a general need to address this issue with Tekton itself? I agree that if Kabanero introduced another orchestration engine as an opinionated alternative, that it may indeed be nice to abstract the "pipeline" or "workflow" template that Kabanero would translate. It might be reasonable to improve Tekton and extend Kabanero.

gireeshpunathil commented 5 years ago

I have been thinking about ways to represent tasks and resources and ways to correlate them in an intuitive way, while not introducing a totally new semantics. Looking at the jenkins' declarative pipeline syntax, I see the verbosity is less, but guess we could do even better:

Here is what I come up with:

Re-use the Dockerfile semantics for pipeline workflow composition, with most of the verbs will carry the same meaning, while reducing the declarations drastically.

For example, a targz pipeline can be abstracted thus:

LABEL targz
FROM ubuntu
ARG input=http://example.com/archive.tar.gz
ENV foo=bar
RUN tar xzvf $input
RUN cat source/file.txt