The ModelMesh framework is a mature, general-purpose model serving management/routing layer designed for high-scale, high-density and frequently-changing model use cases. In essence, it acts as a distributed LRU cache for serving runtime models.
If you are looking at deploying and managing ModelMesh, please use the ModelMesh Serving repo instead, as it hosts the controller for the same. The instructions here are for development purposes only.
runtimeStatus()
- called only during startup to obtain some basic configuration parameters from the runtime, such as version, capacity, model-loading timeoutloadModel()
- load the specified model into memory from backing storage, returning when completemodelSize()
- determine size (mem usage) of previously-loaded model. If very fast, can be omitted and provided instead in the response from loadModel
unloadModel()
- unload previously loaded model, returning when completeregisterModel()
and unregisterModel()
for registering/removing models managed by the clustermm-model-id
or mm-vmodel-id
metadata header (or -bin
suffix equivalents for UTF-8 ids)Prerequisites:
root_prefix
attribute is used as a key prefix for all of the framework's use of etcdFrom an operational standpoint, ModelMesh behaves just like any other homogeneous clustered microservice. This means it can be deployed, scaled, migrated and upgraded as a regular Kubernetes deployment without any special coordination needed, and without any impact to live service usage.
In particular the procedure for live upgrading either the framework container or service runtime container is the same: change the image version in the deployment config yaml and then update it kubectl apply -f model-mesh-deploy.yaml
Sample build:
GIT_COMMIT=$(git rev-parse HEAD)
BUILD_ID=$(date '+%Y%m%d')-$(git rev-parse HEAD | cut -c -5)
IMAGE_TAG_VERSION=0.0.1
IMAGE_TAG=${IMAGE_TAG_VERSION}-$(git branch --show-current)_${BUILD_ID}
docker build -t model-mesh:${IMAGE_TAG} \
--build-arg imageVersion=${IMAGE_TAG} \
--build-arg buildId=${BUILD_ID} \
--build-arg commitSha=${GIT_COMMIT} .