search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.22k stars 89 forks source link

[feature request] aviary model remove #29

Closed brent-anyscale closed 1 year ago

brent-anyscale commented 1 year ago

Provide the ability to remove a model that has been deployed via aviary run - this would allow making tweaks to the underlying yaml file and then redeploying without having to kill ray processes or terminating/restarting the cluster.

Yard1 commented 1 year ago

Will look into it. In the meantime, you can call aviary run with the same model and it will replace the old deployment, or do serve shutdown to kill aviary completely.

mahaddad commented 1 year ago

If aviary run is only meant to be run once with all the desired models and re-running it replaces old deployments then I would also expect that a model not specified is de-provisioned. Example:

If I start with: aviary run --model modelA --model modelB

later, I want to aviary run --model modelB --model modelC

Expected Behavior: modelA is de-provisioned, modelB is re-provisioned, and modelC is provisioned for the first time Encountered Behavior: modelA remains as is, modelB is re-provisioned, modelC is provisioned for the first time

Ultimately it would be a better UX to just individually turn on and off models, but I imagine there's a lot of complexity around managing the cluster that I am not considering.

Yard1 commented 1 year ago

We would like to make it possible to use aviary run multiple times without issues. It is on our to-do list. What you are suggesting, @mahaddad, would be a good way to solve it in the meantime, alongside an aviary remove (aviary stop?) command. I'll look into it on Monday!

avnishn commented 1 year ago

Now if you run aviary run --model a --model b it will tear down the previous models deployment and create a new deployment. There's a warning message with user input to prevent you from doing this on accident.