-
**Describe the bug**
While modified llama3 to llama3.1 as "meta-llama/Meta-Llama-3.1-8B-Instruct".
The model can be managed to download. However it prompt error while sending the input.
> The a…
-
According to the guidebook, I found codes for object detection within only 10 lines, like
#inference.py in jetson-inference
import jetson.utils
import jetson.inference
input = jetson.utils.video…
-
Please comment here any model requests for:
- Models to be added to https://huggingface.co/neuralmagic
- Example scripts within `llm-compressor`
-
Hi, I tried to speed up CNN inference using tensorRT, but the following two problems occurred. Have these issues been addressed?
SuperGlue stuck with warning:
```
[W:onnxruntime:SuperGlueOnnx, …
-
### 请提出你的问题 Please ask your question
指的是不使用TensorRT,只用Paddle Inference,用的是最新的v2.6版本
只在
https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/90f7c86621a2ea2d27617d430a9de88f9f8762d0/c%2B%2B/gp…
-
Now, paddle ERNIE fp32 inference on CPU performance is ass below:
single thread: 251.464 m
20 threads:29.8818 ms
Our goal is to prove that with INT8 real kernel, ERNIE can get the performance gain.…
-
### Describe the issue
I quantized a simple CNN model in Pytorch and converted it to onnx. When I tested the runtime of int8 model and fp32 model on CPU, the int8 model was slower. Here my code:
[Go…
-
I have been trying to run a int8 quantized converted custom keras model on my esp32 cam device but I have been getting discrepancies in outputs. I have so far narrowed it down to a problem with the mu…
-
I tried running IntelAI DLRM model with int8 precision with default int8_configure.json. Could someone clarify if quantization happens each time the inference_performance.sh script is triggered, or if…
-
## Description
For the quantized INT8 model, the inference results are correct under Orin DLA FP16, and the results are also correct under Orin GPU INT8, but the results are completely incorrect un…