Narsil
commented
1 year ago This is something we have in mind, although we're not entirely sure it fits TGI directly.
The feature creep is the biggest reason why we're not sure about it.
The other thing is that supporting multiple Loras will add more memory shifting to a system that's already memory bound, so adding latency, which is the thing we care the most about.
That being said it would be nice indeed to use the reduced memory constraints to serve mulitple models from one.
Code wise it's rather tricky ,because everything is so ingrained as single batch, no padding that having a different model in front for various parts of the batch will require some clever thinking.
RonanKMcGovern
ToddMorrill
garython
Feature request
An increasingly common question is how to support inference for multiple LoRA models running against a single backbone model. What's preventing TGI from implementing a feature like this? I realize there may be a million little reasons but what are the core blockers (e.g. routing, continuous batching, etc.)?
Motivation
The value proposition here is that you can reduce the memory footprint required to serve several models on a single server. You might also get better hardware utilization since demand patterns for different models may complement one another.
Your contribution
I have a decent sense for the optimizations you're applying to the models (e.g. tensor parallelism, kernel fusion, etc.) but less so around routing and continuous batching. I'd have some learning to do before I could contribute a feature like this.