Closed nivibilla closed 6 months ago
+1
+1
+1
+1
+1
+1
+2
+1
+1
+9
+10086
+23333
+120
This could possibly be the solution
+1
The integration is complete. https://docs.vllm.ai/en/latest/models/lora.html
Does this handle multiple loras simultaneously for concurrent calls to different lora adaptors like slora ?
It is exactly S-LoRA with Punica batched gemm kernels. This means concurrent calls of different lora adaptors can be executed in one batch.
https://github.com/S-LoRA/S-LoRA
They call it 'unified paging' and don't require the model to be merged before doing inference. This would be really useful for serving Mixture of Expert models for example or a service that requires multiple different fine-tuned lora adapters based on the same base model.
And needless to say there has been a lot of request for lora deployments