-
python3 detectnet.py --model=peoplenet pedestrians.mp4 pedestrians_peoplenet.mp4
[gstreamer] initialized gstreamer, version 1.14.5.0
[gstreamer] gstDecoder -- creating decoder for pedestrians.mp4
O…
-
### Question
Why is model inference with int8/int4 so much slower than with float16/float32? Aside from decreased memory consumption, shouldn't int8/int4 be faster?![image](https://github.com/haotian…
-
has anyone successfully deployed on orin? what's the infer time like?
-
**Describe the bug**
We conducted tests on OPT/GPTJ/GPT-Neox/BLOOM 7B INT8, these models are all producing garbage outputs on DeepSpeed 0.8.1
OPT model is NCCL communication issue
GPT-…
-
## Description
I recently attempted to utilize INT8 quantization with Stable Diffusion XL to enhance inference performance based on the claims made in a recent [TensorRT blog post](https://developer.…
teith updated
3 months ago
-
### Feature request
It would be immensely useful to have a server-application to serve up HF-Transformer and other Hub models as a service, similar to the how `llama.cpp` bundles the `llama-server`…
-
I test it with batch_num=1, seq_len=128, head_num=5, head_dim=64. It shows "FMHA Inference took 75.82559204 ms, 17.97742325 GFlop/s, 0.01728598 GB/s INT8 average absolute deviation: 1.552685 %". B…
-
I'm curious if anyone has done systematic comparison of original LLAMA inferencing and int8 inferencing.
@tloen great work! much appreciated. I was able to run your example 13B on 16GB GPU (it was …
-
Hi, I see multiple quicsr TFLite models available in the project. The default one is 540p which on GPU, currently takes ~200ms on my phone. Is there a faster one? Any of these maybe for 360p?
![ima…
-
### Describe the issue
Onnxruntime transformers benchmarking is failing for int8 quantized inference. the same is working fine with onnxruntime 1.16.3. I added the error details below.
I found the b…