feat: Client Added for Predibase

ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs

Apache License 2.0

659 stars 113 forks source link

Open bhaba-ranjan opened 8 months ago

bhaba-ranjan commented 8 months ago

Hello,

This pull request adds Ray Client for Predibase.
Predibase is an open-source LLM inference server. It can serve multiple LoRA adapters, in a single deployment.
The pull request targets /generate_stream endpoint which streams token.
API spec can be found here: Predibase Swagger