-
### Checklist
- [X] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related iss…
-
###
I compiled the vllm0.5.4 using the CPU, which does not support AVX512. After compiling, I entered the container and executed the command to start the llama3-8b model.
```text
python3 -m…
-
### 🚀 The feature, motivation and pitch
**Feature motivation:**
[Default pyTorch quantization aware training](https://pytorch.org/docs/stable/quantization.html) uses "fake-quantization" approach. Fo…
-
Hello @dusty-nv
I downloaded peoplnet directly from : https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplenet. These are the contents of the downloaded folder :
labels.txt nvinfer_c…
-
Hey,
I wanted to use peoplenet to detect people in Images. Somehow it is not working correctly. I am using the docker container which is shown in the tutorials. I also tried the image with the "SDD-…
-
### Description
Some inference API providers now support embedding models with each dimension defined as a single bit. For example, the v3 models from Cohere offer this capability. Since we already h…
-
hello!
I build int8 weights:
INFERENCE_PRECISION=float16
WEIGHT_ONLY_PRECISION=int8
MAX_BEAM_WIDTH=4
MAX_BATCH_SIZE=8
checkpoint_dir=whisper_large_v3_weights_${WEIGHT_ONLY_PRECISION}
output_dir…
-
**Describe the bug**
Unable to optimize a model with device- cpu and precision int8. Ending up with KeyError: 'input_model' error
**To Reproduce**
Start with this example: https://github.com/micr…
-
## detail | 详细描述 | 詳細な説明
I was wondering if maybe anyone knows or might have an idea as to why the inference time on the XuanTie 906 (RISC-V) processor is so slow for int8 models?
The image can ea…
-
By using [pytorch-quantization](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/index.html) i was able to create TensorRT engine models that are (almost) fully int8 and…