-
https://github.com/h2oai/h2ogpt/blob/main/docs/TRITON.md
do same for Falcon 7B, then Falcon 40B
-
As indicated by the title, on the main branch, I used 40 threads to simultaneously send inference requests to the in-flight Triton Server, resulting in the Triton Server getting stuck.
The specifi…
-
![image](https://github.com/user-attachments/assets/b2fbbab3-1cc8-4160-b446-b7e09b8089e7)
any suggestions?
11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz 8 16
I…
-
```
root@ttogpu:~# kubectl describe pod triton-inference-server-5b6c7f889c-f54c6
Name: triton-inference-server-5b6c7f889c-f54c6
Namespace: default
Priority: 0
Service …
-
### 🚀 The feature
Implement batched nms!
### Motivation, pitch
My motivation to create this issue was this warning:
```
[W BatchedFallback.cpp:84] Warning: There is a performance drop because w…
-
**Description**
A clear and concise description of what the bug is.
r23.04
```
I0718 11:39:24.385839 1 server.cc:653]
| Model | Version | Status …
-
**Is your feature request related to a problem? Please describe.**
App SDK currently supports inference within the application process itself. This is simple and efficient for some use cases, though …
-
A few options to explore
1. NVIDIA NeMo, TensorRT_LLM, Triton
- NeMo
Run [this Generative AI example](https://github.com/NVIDIA/GenerativeAIExamples/tree/main/models/Gemma
) to build Lora wi…
-
Do you support Exllamav2 backend for the inference that supports exl quants?
The current alternative is vllm but that doesn't support EXL quants. Also, after running a perplexity test, EXL is the b…
-
Hi team, QQ: does `lightseq` support the followings,
- Convert HuggingFace BERT/RoBERTa models to `int8` precision directly
- If yes, can the converted model be exported to ONNX format directly?
- …