search

hltcoe / sandle

Run a large language modeling SANDbox in your Local Environment
Other
7 stars 1 forks source link

Adding, removing backends/backend nodes at runtime #92

Open ccmaymay opened 1 year ago

ccmaymay commented 1 year ago

Starts to bump into issue of load balancing (a more general solution would take load balancing and similar issues into account)