search

google / ml-metadata

For recording and retrieving metadata associated with ML developer and data scientist workflows.
https://www.tensorflow.org/tfx/guide/mlmd
Apache License 2.0
626 stars 147 forks source link

best practices on production deployment? #76

Open RafalSkolasinski opened 4 years ago

RafalSkolasinski commented 4 years ago

I wonder what is a recommended way of deploying mlmd in production?

The only piece in documentation about starting gRPC server I found here:

bazel run -c opt --define grpc_no_ares=true  //ml_metadata/metadata_store:metadata_store_server

Did I miss something?

What would be the best approach to deploy mlmd in Kubernetes? Some example manifests / documentation on this could be very helpful.

paveldournov commented 4 years ago

Please take a look at the Kubeflow Pipelines project, it uses MLMD for tracking lineage of artifacts and jobs.

https://github.com/kubeflow/pipelines

hughmiao commented 4 years ago

@dushyanthsc for deployment guidelines from KFP side.

Jeffwan commented 4 years ago

What's the plan on the multi-tenancy support? I know MLMD doesn't have this concept yet, what's the best practice we should follow?

hughmiao commented 4 years ago

@Jeffwan In the current release, each mlmd-server talks to a single db instance. If the users are allowed to share the same db, then reusing the single server with the released image is fine. When the clients need to store to different db, then multiple server instances are needed at the moment. Would you please elaborate more about the multi-tenancy use case in your deployment to better understand the priorities?

Jeffwan commented 4 years ago

@hughmiao Thanks for the explanation. Sure. Let me collect more requirements internally and come back with a concrete summary and then we can have some discussion.

htahir1 commented 2 years ago

I'm also looking for some guidance here. How exactly do we deploy MLMD in production? What is the proper guidance here? Is there a docker image that we can simply run without going through Bazel? Any tips might be helpful :)