search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
995 stars 231 forks source link

autoscaling with --model-store #903

Closed xxxpsyduck closed 4 years ago

xxxpsyduck commented 4 years ago

It seems like autoscale only works with --models but not --model-store

xxxpsyduck commented 4 years ago

When using --model-store, I'm forced to use Management API to register and scale model. I just want to use one command only to start server, register and scale model. Is it possible to use just one command and then all the models will be ready for reference request?

vdantu commented 4 years ago

when you configure --model-store try to use load_models=ALL in config.properties .. That should technically load all the models in the --model-store and scale it to num-gpu's or num-cpu's

vdantu commented 4 years ago

@xxxpsyduck : Please do let us know if this works for you :) .. This will essentially let you start MMS with a single command, and all the models in the model-store will be loaded all at once and scaled for inference requests.

xxxpsyduck commented 4 years ago

it works! thanks!