search

mosecorg / mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
https://mosec.readthedocs.io/
Apache License 2.0
746 stars 51 forks source link

unload model automatic #393

Closed lfxx closed 1 year ago

lfxx commented 1 year ago

Describe the feature

can we add a function which can unload model if there is no request received within one day.And model will be loaded again if new request be receive.

Why do you need this feature?

This is very useful on GPU memory urgent server.

Additional context

No response

lfxx commented 1 year ago

I have implented this on my own server framework.But i still hope mosec can have this funcion.

gaocegege commented 1 year ago

Thanks for the issue.

By the way, could you tell me in what scenarios Mosec is being used in your case, and how do you know mosec?

lfxx commented 1 year ago

Thanks for the issue.

By the way, could you tell me in what scenarios Mosec is being used in your case, and how do you know mosec?

I learned about mosec from a blogger named oldpan, and I will use mosec to build an AI service for frontend to use. i hope mosec can add this funtion soon since it is a bit of complex for me to implement this on mosec cause i am not a rust expert.