Closed gyliu513 closed 4 years ago
Thanks @gyliu513. This is a very interesting proposal. We need to compare the feature set with Argo. I am not sure that it would make sense to support two pipeline resources if the feature set if similar. What do you think?
@vicaire Seems the goal of https://github.com/tektoncd/pipeline is to build a K8s-native Pipeline resource, tekton pipeline is also a project of CDF https://cd.foundation/ , so the tekton pipeline seems deserve to check more ;-)
Sounds good. We will have a look at Tekton.
Tekton is focusing on CI/CD scenarios while Argo and KF Pipelines have bigger scope and are more general-purpose.
@Ark-kun , would you like to share a little more of the bigger scope of KF/Argo?, i'm evaluating tekton/KF/Argo for our next continuous deployment tool.
@vicaire Tekton is becoming the CI/CD backbone behind our cloud and even in OpenShift. Would like to see if you have given this more thought, and this might also be an opportunity to decouple KFP from Argo to some extent cc @paveldournov @IronPan
@animeshsingh we were discussing Tekton PIpelines today - primarily as they are the backend to Jenkins X, and part of the CD.Foundation.
It's still early stages but I would be interested to learn more about how you are using Tekton in your cloud.
Furthermore, I personally see there is may be some opportunity within KFP, as currently I feel there is quite a lot of ambiguity around CI/CD vs ETL logic within individual KF Pipelines.
I think it would be interesting to explore even a conceptual separation between the use of KF Pipelines for ETL (ML Training / Batch processing) vs the CI/CD of trained pipelines.
I agree that having yet another service added to the KF Package is not desired, but exploring Tekton pipelines itself and even decoupling KFP from Argo could provide interesting insights on this conceptual separation of ETL vs CI/CD.
Perhaps even having something that would remove the CI/CD from the KFP all together would even help on this separation of concerns alltogether, which means that KFP could be left untouched.
What are your thoughts on this?
@axsaucedo @animeshsingh I thought someone at the MLOps Sig meeting someone mentioned that they were working on a KFP DSL compiler that would execute on Tekton?
Does KFP have a well defined resource model? Would Tekton make a good resource model for pipelines?
There were some plans to split the DSL compiler in two parts - the first creates a logical pipeline (a pipeline/graph component) and the second stage that converts the graph component to Argo workflow. This would make it easier to support more runners and orchestrators as they only need to implement the second part.
What are the key differences between Argo and Tekton? Does Tekton help you pass data between tasks? (this is a core feature that Argo supports).
Here's my limited understanding of Tekton's model.
@bobcatfish and other tekton experts might be able to shed more light on Tekton
found a good discussion on tektoncd vs argo https://github.com/tektoncd/pipeline/issues/644#issuecomment-475032948
If you are interested we created this spreadsheet a while ago which compares some of the Tekton Pipeline goals with existing CI/CD tools (though it compares Argo, not Argo CI or CD). (note this is shared with members of https://groups.google.com/forum/#!forum/knative-dev - since we used to be part of knative! - you'll have to join the group to see the doc)
Some notes.... Tekton seems more focused to source-based pipelines. Lots of similarities, the below were my takeaways:
There is also this announcement: https://www.weave.works/blog/argo-flux-join-forces
This space seems rapidly developing.
@Ark-kun there is some work going on for kfp-tekton compiler here, curently being driven by us, but we are hoping we can others join in the effort.
Design doc and specifications: http://bit.ly/kfp-tekton KFP-Tekton-Argo comparision: https://bit.ly/kfp-tekton-argo-comparision Github repo: https://github.com/kubeflow/kfp-tekton CD Foundation MLOps Sig: https://cd.foundation/blog/2020/02/11/announcing-the-cd-foundation-mlops-sig/
The work is being driven under the MLOps Sig there. Its a biweekly 30 min sync, so would be great if you can subscribe and join
There were some plans to split the DSL compiler in two parts - the first creates a logical pipeline (a pipeline/graph component) and the second stage that converts the graph component to Argo workflow. This would make it easier to support more runners and orchestrators as they only need to implement the second part.
Any further work on your side? If you want, we should start some internal discussions
cc @jlewi
@Ark-kun coming back to it. Would be great to syncup on the above cc @jessiezcc @paveldournov
Duplicate of #3647
Tekton pipeline plan to be a k8s-native piepline resource, so any plan for kubeflow pipeline integrate with it?
/cc @jlewi @jinchihe @hougangliu