simra
commented
1 year ago This makes sense. The two main cost drivers are redis and the prediction API which is too big an image for a B1 plan. The celery worker is already running in the same container as the flask service and all it does is receive messages and put POST requests to the API, writing the result to the SQL back-end. We could locally run the celery broker (I tried rabbitmq instead of redis but had issues), and trigger the prediction service to spin up only when we need it. I had a thought to just migrate the prediction api to azure functions but I'm not familiar enough to know if that would work like I think it would.
On Sun, Jan 8, 2023 at 2:42 AM Chris Gough @.***> wrote:
Is this about making the ML model deployments more cost-effective, or the rest of it (front-end, Flask microservice, celery, DB, redis)? If it's about the prediction endpoint...
What if we put the celery worker and prediction endpoint in the same worker (docker-composition / host), and had some lightweight supervisor process that shut it down when there was no workload (i.e. when the queue had been empty for a minute), and woke it up again when the queue had been non-empty for longer than some threshold (e.g. 3 minutes, assuming it takes less than 2 minutes to boot up). So when the supervisor woke the worker up, it would both have the prediction endpoint and the celery worker munching through the queue and getting predictions done until there were no more, then it would go to sleep. Yes I'm sure there is a K8 way to do this, but maybe an Azure native way too? Something for the cloud guys to look at...
This way, if one of us had a cheap but unreliable celery/prediction worker (i.e. on a workstation) the queue would seldom if ever be "non-empty for some time", unless things were getting really busy with users, so the supervisor would seldom need to wake up the worker and start spending real money. Unless there was some outage of the workstation, in which case (if anyone was actually using the app) the supervisor would pick up the slack.
— Reply to this email directly, view it on GitHub https://github.com/Tech4Tracing/CartridgeOCR/issues/107#issuecomment-1374796985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWPNL6OEFEIGGBUBHCLNTDWRKKYRANCNFSM6AAAAAATLNWYME . You are receiving this because you authored the thread.Message ID: @.***>
monkeypants
Is this about making the ML model deployments more cost-effective, or the rest of it (front-end, Flask microservice, celery, DB, redis)? If it's about the prediction endpoint...
What if we put the celery worker and prediction endpoint in the same worker (docker-composition / host), and had some lightweight supervisor process that shut it down when there was no workload (i.e. when the queue had been empty for a minute), and woke it up again when the queue had been non-empty for longer than some threshold (e.g. 3 minutes, assuming it takes less than 2 minutes to boot up). So when the supervisor woke the worker up, it would both have the prediction endpoint and the celery worker munching through the queue and getting predictions done until there were no more, then it would go to sleep. Yes I'm sure there is a K8 way to do this, but maybe an Azure native way too? Something for the cloud guys to look at...
This way, if one of us had a cheap but unreliable celery/prediction worker (i.e. on a workstation) the queue would seldom if ever be "non-empty for some time", unless things were getting really busy with users, so the supervisor would seldom need to wake up the worker and start spending real money. Unless there was some outage of the workstation, in which case (if anyone was actually using the app) the supervisor would pick up the slack.