search

predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
https://loraexchange.ai
Apache License 2.0
1.89k stars 125 forks source link

vision language model support #179

Open 7uk3y opened 5 months ago

7uk3y commented 5 months ago

Feature request

The developments in the robotics community around RT-2 show a lot of potential for VLMs but the hardware constraints for small developers makes it difficult to deploy RT-2 level performance with off the shelf hardware suitable for robotics. A potential path forward would be using very small models that have a library of loras to access when the environment calls for them. Lorax looks like it might be a way forward for smaller devs and robot enthusiasts to use and experiment with VLM architectures

Motivation

robots as a hobby

Your contribution

testing and refining available models

tgaddair commented 5 months ago

Thanks for the filing this issue! I think this shouldn't be too hard to add support for some popular ones like LLaVA.

pbarker commented 5 months ago

Hey @tgaddair I would be up to give this a try, could you just give me a basic rundown of what you think needs to be done here to save me some head scratching