Open anamariaUIC opened 5 months ago
I'm using slurm and srun to acreate an enroot jail of vllm, and then using privgpt to connect to vllm as "openailike" and this seems to work well enough.
Privategpt could be a little snapper, but I am still getting like 10 tokens/sec as put. Initial queries take around 2 seconds.
Thank you for that. Would you be available to chat a bit further abotu that?
On Apr 11, 2024, at 4:43 PM, icsy7867 @.**@.>> wrote:
CAUTION: External Sender
I'm using slurm and srun to acreate an enroot jail of vllm, and then using privgpt to connect to vllm as "openailike" and this seems to work well enough.
Privategpt could be a little snapper, but I am still getting like 10 tokens/sec as put. Initial queries take around 2 seconds.
— Reply to this email directly, view it on GitHubhttps://github.com/zylon-ai/private-gpt/issues/1805#issuecomment-2050604021, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUHBH5CCKQVZNHQU5UIYHJLY437YLAVCNFSM6AAAAABFNFVZOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQGYYDIMBSGE. You are receiving this because you authored the thread.Message ID: @.***>
This email originated from outside the University of Illinois System. Use caution when replying, clicking links, or opening attachments. DO NOT reply to any requests asking you to reply from a personal account or SMS.
Hello,
I installed privateGPT using:
My NVIDIA GPU shown in attach.
One query takes one hour to be answered please advise.
Thanks