Starting a service starved for resources can happen while proxy is stopping

perk11 / large-model-proxy

Large Model Proxy is designed to make it easy to run multiple resource-heavy Large Models (LM) on the same machine with limited amount of VRAM/other resources. It listens on a dedicated port for each proxied LM, making them always available to the clients connecting to these ports.

GNU General Public License v2.0

46 stars 3 forks source link

Starting a service starved for resources can happen while proxy is stopping #10

Open perk11 opened 1 month ago

perk11 commented 1 month ago

After interrupt signal is received, a service that previously had no resources to start can get started after resources are freed by the interrupt command.

The proxy should instead close connections for services that hasn't yet started

perk11 commented 1 month ago

This should be fixed in b67f4b04aa4a9ce638ef1909b73eb3aeec3f66e4, but I will keep this open until there is a test