Open yyChen233 opened 3 months ago
I met a bug when I use torch-tensorrt to quantize a YOLOv6 model into fp8_e4m3. I use the code in vgg_fp8_code , but I met the bug report as below:
E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Error while creating guard: E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Name: "type(L['self'].neck.Bifusion0.upsample.upsample_transpose).mro[4].forward.defaults[0]" E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Source: local_nn_module E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Create Function: CONSTANT_MATCH E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Guard Types: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Code List: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Object Weakref: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Guarded Class Weakref: None Traceback (most recent call last): File "/mnt/d/LM/YOLOv6/test_tensorrt.py", line 96, in exp_program = torch.export.export(model, (input_tensor,)) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/init.py", line 174, in export return _export( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 635, in wrapper raise e File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 618, in wrapper ep = fn(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/exported_program.py", line 83, in wrapper return fn(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 860, in _export gm_torch_level = _export_to_torch_ir( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 347, in _export_to_torch_ir gm_torchlevel, = torch._dynamo.export( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1311, in inner result_traced = opt_f(args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn return fn(args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors return callback(frame, cache_entry, hooks, frame_state, skip=1) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert return _compile( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/contextlib.py", line 79, in inner return func(args, kwds) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 703, in _compile raise InternalTorchDynamoError(str(e)).with_traceback( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, **kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in compile_inner check_fn = CheckFunctionManager( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1048, in init guard.create(builder) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_guards.py", line 249, in create return self.create_fn(builder, self) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 501, in CONSTANT_MATCH val = self.get(guard.name) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 284, in get return eval(name, self.scope, CLOSURE_VARS) File "", line 1, in torch._dynamo.exc.InternalTorchDynamoError: 'NoneType' object is not subscriptable
I wonder why this bug happened. Are there some unsupported architectures in YOLOv6 for fp8 quantization?
My code is copied from the examples of VGG-fp8 like this:
` eval = Evaler(data=data, batch_size=4, img_size=640, conf_thres=0.4, device=device, shrink_size=None, plot_curve=False, half=False) model = eval.init_model(None, weights, task)
def eval_func(model): model.eval() eval2 = Evaler(data=data, batch_size=2, img_size=640, conf_thres=0.4, device=device, shrink_size=None, plot_curve=False,) eval2.stride = int(model.stride.max()) dataloader2 = eval2.init_data(None, task) pred_result, vis_outputs, vis_paths = eval2.predict_model(model, dataloader2, task) map50, map = eval2.eval_model(pred_result, model, dataloader2, task) del eval2 return map50 quant_cfg = mtq.FP8_DEFAULT_CFG # PTQ with in-place replacement to quantized modules mtq.quantize(model, quant_cfg, forward_loop=eval_func) # model has FP8 qdq nodes at this point input_img = torch.randn([1, 3, 640, 640]) with torch.no_grad(): with export_torch_mode(): # Compile the model with Torch-TensorRT Dynamo backend input_tensor = input_img.cuda() exp_program = torch.export.export(model, (input_tensor,)) trt_model = torchtrt.dynamo.compile( exp_program, inputs=[input_tensor], enabled_precisions={torch.float8_e4m3fn}, min_block_size=1, debug=False, ) print(trt_model)`
Build information about Torch-TensorRT can be found by turning on debug messages
conda
pip
libtorch
Seems like an issue with the dynamo export procedure. The main pytorch repo might be a better source of help.
Bug Description
I met a bug when I use torch-tensorrt to quantize a YOLOv6 model into fp8_e4m3. I use the code in vgg_fp8_code , but I met the bug report as below:
I wonder why this bug happened. Are there some unsupported architectures in YOLOv6 for fp8 quantization?
To Reproduce
My code is copied from the examples of VGG-fp8 like this:
` eval = Evaler(data=data, batch_size=4, img_size=640, conf_thres=0.4, device=device, shrink_size=None, plot_curve=False, half=False) model = eval.init_model(None, weights, task)
Environment
conda
,pip
,libtorch
, source): pip