search

predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
https://loraexchange.ai
Apache License 2.0
1.86k stars 126 forks source link

Invoke API based external models #321

Open sekhar-hari opened 3 months ago

sekhar-hari commented 3 months ago

Hi -

How do I call OpenAI GPT (say gpt-4) and Google Gemini models through LoRAX? An example code snippet would really help.

Thanks, Sekhar H.

tgaddair commented 3 months ago

Hey @sekhar-hari, LoRAX does not support querying external models. That would be something better served by an API gateway like PortKey.