search

iterative / mlem

🐶 A tool to package, serve, and deploy any ML model on any platform. Archived to be resurrected one day🤞
https://mlem.ai
Apache License 2.0
718 stars 44 forks source link

Deploying models with seldon-core #20

Open aguschin opened 3 years ago

aguschin commented 3 years ago

@DavidGOrtega suggested on slack:

Hey guys. Reviewing OpenMlOps they are exposing the models via Seldon and Ambassador API Gateway. This is similar to the QuaroML stack that I did and a very convenient and easy way to expose and of course scale the models. Maybe we can introduce MLEM inside the TPI in conjunction with Consul

We need to get back to this discussion to shape the vision for MLEM deployment part. It's better to do as soon as we are finished with the closed alpha release.

aguschin commented 2 years ago

Maybe we can introduce MLEM inside the TPI in conjunction with Consul

@DavidGOrtega, I'm getting back to your comment since @mike0sv is working on Deployment 2.0 in MLEM. Could you please elaborate on your thought? How do you see this?

Beside that, could MLEM integrate with TPI to deploy to AWS EC2 or GCP?

aguschin commented 2 years ago

@casperdcl @0x2b3bfa0 @DavidGOrtega

Also, we're now implementing deployment to Sagemaker. To do that e2e we need to provision some AWS resources for the user (Roles, Policies, etc). Can we use TPI to do that? Or do you have plans to implement an option to provision this?

casperdcl commented 2 years ago

Looks like 3 different feature requests:

  1. "exposing the models via Seldon and Ambassador API Gateway [...] similar to the QuaroML stack [...] introduce MLEM inside the TPI in conjunction with Consul"
    • not sure I follow. Is this a new cloud feature request on the TPI repo @DavidGOrtega?
  2. "MLEM integrate with TPI to deploy to AWS EC2 or GCP"
    • sure, TPI can: provision an instance, upload workdir, run a script on the instance, auto-recover from spot interruptions (restores wokrdir & re-runs script), auto-cleanup on script exit, download workdir.
    • requirement: provide cloud (AWS/Azure/GCP) credentials via environment variables^1
    • you can use docker^2
    • you can use avoid having to config/expose service ports on AWS by using a (free) port forwarding service^3
    • you can use the Python wrapper (pip install tpi) which auto-detects OS, downloads/caches terraform binaries, installs TPI, and even has a Python API so you don't explicitly run CLI
  3. "deployment to Sagemaker. To do that e2e we need to provision some AWS resources for the user (Roles, Policies, etc)"