Allow kubeflow to render graph when using pipelineRef or taskRef

[mat0pad]

/kind feature

Description: The Kubeflow dashboard fails to render all custom CRDS using pipelineRef and taskRef. This makes it differcult to reuse existing Tasks or Pipelines created on the cluster.

Goal: Allow rendering of dashboard graph when using pipelineRef and taskRef

[fenglixa]

@mat0pad Do you have use case for this issue? This is the design of kubeflow UI. which need to import one object and render graph based on it, I am not say it is not doable. but much more complex. That's why we pushed tekton community to support pipelineSpec and taskSpec for custom task feature.

[Tomcli]

Hi @mat0pad, do you have a use case where you need to use pipelineRef and taskRef? We are only planning to support taskRef for recursion use case only. The KFP-Tekton UI is mainly for rendering generated pipelines from the KFP-Tekton SDK.

[mat0pad]

Hi @fenglixa @Tomcli I see multiple use cases and benefits for this:

• Having multiple pipelines using the same tasks/pipelines requires a mantainer to update all taskSpecs or pipelineSpecs manually if a change is needed. Say each pipeline have a task that clones the source code then if the version control changed, hosting service or protocol then the mantainer would need to update all pipelines manually. Another example could be a cleanup job that runs using finally which needs to use another image. By allowing taskRef and pipelineRef to be used we make it much more mantainable for production grade systems and a mantainer could just update a clustertask or task directly.
• Less duplication in pipelines and tasks
• Support for using community tasks (i.e. https://hub.tekton.dev/)
• Decouple the data sicence from the complex tasks that can be predefined

I come from IBM (CIC) and on one of our larger projects we're actually considering moving to our own solution with tekton, tekton dashboard, knative and an artifact storage service because of the lacking feature.

[Tomcli]

Thanks @mat0pad, I think your use case is more suited for the Tekton dashboard. Kubeflow Pipeline dashboard is fundamentally different because all the information in the dashboard except pods and events are pulled from a relational database where we archive all the Tekton pipelineRun and metadata. This way we are not limited to the Kubernetes etcd service and able to store tens of thousands of jobs, experiments, and pipelines without adding too much overhead to the Kubernetes API.

Please take a look at the KFP design over here. The fundamental idea of KFP is allow users to design their pipelines using the Python SDK and version control the task using a component.yaml. This way there's very minimum interaction with etcd and users are able to carry their pipelines and tasks across multiple clusters. https://www.kubeflow.org/docs/components/pipelines/overview/pipelines-overview/#architectural-overview

This also applies to the Argo backend where the Kubeflow Pipeline dashboard will not render the Argo workflowTemplate CRDs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.