search

aws / sagemaker-inference-toolkit

Serve machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.
Apache License 2.0
379 stars 82 forks source link

Config parsing can be improved #16

Open ericangelokim opened 4 years ago

ericangelokim commented 4 years ago

https://github.com/aws/sagemaker-inference-toolkit/blob/master/src/sagemaker_inference/model_server.py#L42

This code here can be improved to add a config file, and not just use the mms default or env vars that are parsed.

The utils can include support to merge/override config files, instead of appending string values to write to file.

laurenyu commented 4 years ago

thanks for the suggestion!

arvarik commented 4 years ago

+1

Currently it seems like only these 4 variables are able to be configured.

Having the ability to define your own config.properties would be nice (job_queue_size is particularly useful to me)

sanjams2 commented 4 years ago

For users less keen on updating config files, it would also be nice if these custom config values could be passed to start_model_server. If there is worry these values would be lost on restart, then Inference Toolkit could take care of writing these custom values out to a file on disk that can be re-used.

ziweijiang commented 2 years ago

+1 When I'm trying to do inference for a file larger than 9M, the MMS will get stuck. And I found a solution in MMS github issue, configuring max_request_size. But I'm not able to do it. It would be really helpful it custom config file is supported.

neo commented 7 months ago

I come from here also looking to configure the job_queue_size: https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-troubleshooting.html