search

elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.83k stars 343 forks source link

Evaluate/add support for Kubeflow component registry hosts #2733

Open ptitzler opened 2 years ago

ptitzler commented 2 years ago

Is your feature request related to a problem? Please describe. An upcoming release (Q2 2022) of Kubeflow Pipelines will introduce support for component registries. We need to evaluate how Elyra can support those registries via its catalog connector infrastructure. Details will be added to this issue as they emerge.

Related: https://github.com/elyra-ai/examples/issues/107 (MLX will implement the registry host API).

thesuperzapper commented 2 years ago

@ptitzler can you send a link to where you saw something about Kubeflow Pipelines component registries?

ptitzler commented 2 years ago

I learned about registries in a recent conference talk from the MLX team. They are planning to implement the registry host API in a future release.

thesuperzapper commented 2 years ago

@ptitzler I think that is pretty specific to MLX, and while cool, not everyone will be using MLX, so we will need to keep pushing multiple solutions for component registries.

Also, I think if we can find a way to expose a "public" interface for "generic" Elyra components (possibly like proposed in https://github.com/elyra-ai/elyra/issues/2699), then we would need an Elyra-specific component catalog API for these generic components, as they would be higher-level than Kubeflow or Airflow.

We might even be able to make something like DockerHub for generic Elyra components, but who knows.

ptitzler commented 2 years ago

@ptitzler I think that is pretty specific to MLX, and while cool, not everyone will be using MLX, so we will need to keep pushing multiple solutions for component registries.

Right! We are continuing to support (and improve) the existing catalog connector API based on feedback we receive from contributors who've to to implement one. Since connector development requires relatively few insights into the innner workings of Elyra, we (the maintainers) are relying on the extended community to contribute connector implementations.