search

InftyAI / llmlite

🌵 A library helps to communicate with all kinds of LLMs consistently.
MIT License
6 stars 2 forks source link

batching #45

Closed Jerry-Kon closed 6 months ago

Jerry-Kon commented 6 months ago

chatglm has been supported llama backend (hf, vllm) exits bug

Jerry-Kon commented 6 months ago

add batching function to vllm backend, solved the bugs

kerthcet commented 6 months ago

Also please solve the conflict.

kerthcet commented 6 months ago

Can you fix this? I'm waiting for this feature.

kerthcet commented 6 months ago

close in favor of https://github.com/InftyAI/llmlite/pull/50