search

NVIDIA-AI-IOT / yolo_deepstream

yolo model qat and deploy with deepstream&tensorrt
Apache License 2.0
549 stars 139 forks source link

RuntimeError: range of [0, nan] is not finite #62

Open RonaldYuren opened 3 months ago

RonaldYuren commented 3 months ago

I made a few modification for yolov7-tiny ( Use a new kind of activation function )， But there comes a new problem just like below.

######## below are the error message: Traceback (most recent call last): File "/home/si2/YuRen/yolov7/yolov7_qat/scripts/qat.py", line 319, in cmd_quantize( File "/home/si2/YuRen/yolov7/yolov7_qat/scripts/qat.py", line 161, in cmd_quantize quantize.calibrate_model(model, train_dataloader, device) File "/home/si2/YuRen/yolov7/yolov7_qat/scripts/quantize.py", line 292, in calibrate_model collect_stats(model, dataloader, device, num_batch=num_batch) File "/home/si2/YuRen/yolov7/yolov7_qat/scripts/quantize.py", line 273, in collect_stats output = model(imgs) ^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1527, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/YuRen/yolov7/models/yolo.py", line 606, in forward return self.forward_once(x, profile) # single-scale inference, train ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/YuRen/yolov7/models/yolo.py", line 632, in forward_once x = m(x) # run ^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1527, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/YuRen/yolov7/models/common.py", line 129, in fuseforward return self.act(self.conv(x)) ^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1527, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/pytorch_quantization/nn/modules/ quant_conv.py", line 125, in forward quant_input, quant_weight = self._quant(input) ^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/pytorch_quantization/nn/modules/ quant_conv.py", line 85, in _quant quant_input = self._input_quantizer(input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/torch/nn/modules/module.py", lin e 1527, in _call_impl return forward_call(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/pytorch_quantization/nn/modules/ tensor_quantizer.py", line 338, in forward self._calibrator.collect(inputs) File "/home/si2/anaconda3/envs/YuRen1/lib/python3.11/site-packages/pytorch_quantization/calib/histo gram.py", line 109, in collect self._calib_hist = torch.histc(x, bins=self._num_bins, min=0, max=x_max) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: range of [0, nan] is not finite

########

How do I solve this question about the "Nan" value ?