search

elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.81k stars 336 forks source link

Elyra looks amazing. Interest in helping us develop Flyte pipelines integration? #753

Open kumare3 opened 4 years ago

kumare3 commented 4 years ago

Elyra looks great and is absolutely a missing piece. At Flyte we provide the infrastructure to run and orchestrate data and ML workflows at scale. We would love to explore the integration of Elyra with Flyte (for both - batch job and pipeline execution)

lresende commented 4 years ago

@kumare3 thanks for your interest in Elyra and willing to work with the community to add support for Flyte. It would great to enable Elyra to run pipelines into Flyte, and looking at a high level, it would probably be a similar effort to what we did for Kubeflow Pipelines and Apache Airflow as Flyte seems to follow a similar pattern and have similar SDKs

image

A good place to start looking would be: https://github.com/elyra-ai/elyra/blob/master/elyra/pipeline/processor.py https://github.com/elyra-ai/elyra/blob/master/elyra/pipeline/processor_kfp.py

And some direct integration with KFP is located on its own module, but we are looking into refactoring it to leverage container inits, etc https://github.com/elyra-ai/kfp-notebook

We have our community dev meetings on Thursday's morning at 9 AM PST, maybe you could come to one of these and demo Flyte for the community, we could also schedule a more specific call or use this issue to discuss details/questions your team might have.

kumare3 commented 3 years ago

@lresende sorry for the late response. This sounds awesome. I would Love to demo. Maybe a week from now - Thursday 30th July?

lresende commented 3 years ago

@kumare3 Please let us know (or just show up) if you still want to demo it on our Thursday's meetings. I am also available for any technical discussion around this.

kumare3 commented 3 years ago

hi - ohh this fell off my radar as we got a little busy on our end. We are also working on a major overhaul of our python sdk, but a demo is a great idea to start collaboration. I will try to message you on "gitter"

lresende commented 3 years ago

Additional info about how Flyte runs notebooks jobs: https://github.com/lyft/flytesnacks/tree/master/cookbook/recipes/notebook_tasks