search

onepanelio / onepanel

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
https://docs.onepanel.ai/
Apache License 2.0
716 stars 69 forks source link

Support for Passing Data among Argo Jobs #612

Open kaushikb11 opened 3 years ago

kaushikb11 commented 3 years ago

Feature Request

Let’s consider I have triggered an Argo Workflow, it is a DAG of multiple Jobs. Jobs A and B are running parallelly & we want data to pass from Job A to Job B. Example: Job B does Tracking & Job A does OD with Bounding Boxes as outputs. Right now on our platform, we pass data through queues and let the Jobs polls on it.

It would help in extending Onepanel’s Workflow (Argo) capability for building Video Applications or Batch Processing Jobs.

issue-label-bot[bot] commented 3 years ago

Issue-Label Bot is automatically applying the label kind/enhancement to this issue, with a confidence of 0.85. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

rushtehrani commented 3 years ago

Related Argo issues https://github.com/argoproj/argo-workflows/issues/2729 https://github.com/argoproj/argo-workflows/issues/2551

sarabala1979 commented 3 years ago

@rushtehrani are you using Argo workflow? would you like to add onepanel in argo users.md?

sarabala1979 commented 3 years ago

I would like to here the more details for the Stream workflow usecase. Basically, I am waiting for real-world usecases to start the workflow.

rushtehrani commented 3 years ago

are you using Argo workflow? would you like to add onepanel in argo users.md?

@sarabala1979 Yes we are and I will absolutely send a PR to add us to users.md.

I would like to here the more details for the Stream workflow usecase. Basically, I am waiting for real-world usecases to start the workflow.

@kaushikb11 has a great use case with videos. It's basically a combination of this issue and https://github.com/onepanelio/core/issues/613.

I am also exploring other related use cases and will provide more information in https://github.com/argoproj/argo/issues/2729 based on my findings.