-
ttok adds a 20 second delay when using llm-gpt4all in offline compared to online tested on mistral-7b-instruct-v0.
This is easily seen when looking at the CPU usage after asking the model a questio…
-
### Describe the issue
> [!TIP]
> ## Want to get involved?
> We'd love it if you did! Please get in contact with the people assigned to this issue, or leave a comment. See general contributing ad…
-
For example i want to run an endpoint which is running on intel/amd/nvidia instances which is generally available. ie, instances which are not available on Sagemaker but available on AWS.
-
**Is your feature request related to a problem? Please describe.**
We are able to infer the recommendation by the qualification tool but the recommendation is based on vague GPUs.
In recent experi…
-
**Is your feature request related to a problem? Please describe.**
Currently we only support `retrieve_online_documents`. We should add a way to allow for vector search for historical retrieval so us…
-
**Is your feature request related to a problem? Please describe.**
Have a good vacation!
Most of the time dealing with LLM is dealing with limited context window size (especially in text summariza…
-
### 是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
- [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
### 该问题是否在FAQ中有解答? | Is there an existing ans…
-
## 🚀 Feature
It would be great if you could support whisper voice recognition on the phone with for example [whisper.cpp](https://github.com/ggerganov/whisper.cpp). Maybe in cooperation with [Hello T…
-
```
File "/opt/homebrew/lib/python3.10/site-packages/chat_with_mlx/app.py", line 166, in chatbot
response = client.chat.completions.create(
File "/opt/homebrew/lib/python3.10/site-packages/…
-
The engine is ok using python to run offline inference with trt-llm.
But when I use triton to run it, it complains like following.
Why is this? The triton server uses more memory than TRT-LLM of…