search

argoproj-labs / hera

Hera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo. ⭐️ Remember to star!
https://hera.rtfd.io
Apache License 2.0
550 stars 105 forks source link

Argo for model serving #1126

Closed sachinruk closed 1 month ago

sachinruk commented 1 month ago

First of all, thank you to the creaters and maintainers of Hera. This has been such a godsend.

Was wondering if its possible to use argo for model serving as opposed to training/ batch jobs. It is possible to deploy a k8s app that will host a dockerised fastai endpoint (that can autoscale according to requests).

I'm hoping that given the underlying k8s architecture there is a way to make a persistent (say fastapi) endpoint with Hera. If so how would I do that?

TIA.

agilgur5 commented 1 month ago

For reference, this was x-posted to #argo-workflows Slack, where I responded and said that while Argo can quite easily fit a batch serving model, for API-driven real-time serving, KServe/Seldon/etc are a better fit and I have used them respectively for batch vs real-time inference.

You can also use Workflows to create Deployments or InferenceServices (i.e. your MLOps pipelines), but CD may suffice for that too.

In short, there are purpose built tool stacks for each of these things, although you can certainly mix some parts together.

Also this sounds like it should've been a Discussion rather than an issue.