search

pytorch / TensorRT

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
https://pytorch.org/TensorRT
BSD 3-Clause "New" or "Revised" License
2.57k stars 350 forks source link

🐛 [Bug] Encountered bug : torch._dynamo.exc.InternalTorchDynamoError: 'NoneType' object is not subscriptable #3025

Open yyChen233 opened 3 months ago

yyChen233 commented 3 months ago

Bug Description

I met a bug when I use torch-tensorrt to quantize a YOLOv6 model into fp8_e4m3. I use the code in vgg_fp8_code , but I met the bug report as below:

E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Error while creating guard: E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Name: "type(L['self'].neck.Bifusion0.upsample.upsample_transpose).mro[4].forward.defaults[0]" E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Source: local_nn_module E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Create Function: CONSTANT_MATCH E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Guard Types: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Code List: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Object Weakref: None E0720 18:37:39.430194 140203970340672 torch/_guards.py:251] [0/0] Guarded Class Weakref: None Traceback (most recent call last): File "/mnt/d/LM/YOLOv6/test_tensorrt.py", line 96, in exp_program = torch.export.export(model, (input_tensor,)) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/init.py", line 174, in export return _export( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 635, in wrapper raise e File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 618, in wrapper ep = fn(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/exported_program.py", line 83, in wrapper return fn(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 860, in _export gm_torch_level = _export_to_torch_ir( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/export/_trace.py", line 347, in _export_to_torch_ir gm_torchlevel, = torch._dynamo.export( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1311, in inner result_traced = opt_f(args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn return fn(args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors return callback(frame, cache_entry, hooks, frame_state, skip=1) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert return _compile( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/contextlib.py", line 79, in inner return func(args, kwds) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 703, in _compile raise InternalTorchDynamoError(str(e)).with_traceback( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, **kwargs) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in compile_inner check_fn = CheckFunctionManager( File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1048, in init guard.create(builder) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_guards.py", line 249, in create return self.create_fn(builder, self) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 501, in CONSTANT_MATCH val = self.get(guard.name) File "/home/cyy/anaconda3/envs/yolo_quantize/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 284, in get return eval(name, self.scope, CLOSURE_VARS) File "", line 1, in torch._dynamo.exc.InternalTorchDynamoError: 'NoneType' object is not subscriptable

I wonder why this bug happened. Are there some unsupported architectures in YOLOv6 for fp8 quantization?

To Reproduce

My code is copied from the examples of VGG-fp8 like this:

` eval = Evaler(data=data, batch_size=4, img_size=640, conf_thres=0.4, device=device, shrink_size=None, plot_curve=False, half=False) model = eval.init_model(None, weights, task)

def eval_func(model):
    model.eval()
    eval2 = Evaler(data=data, 
              batch_size=2,
              img_size=640,
              conf_thres=0.4, 
              device=device, 
              shrink_size=None,
              plot_curve=False,)
    eval2.stride = int(model.stride.max())
    dataloader2 = eval2.init_data(None, task)
    pred_result, vis_outputs, vis_paths = eval2.predict_model(model, dataloader2, task)
    map50, map = eval2.eval_model(pred_result, model, dataloader2, task)
    del eval2
    return map50

quant_cfg = mtq.FP8_DEFAULT_CFG
# PTQ with in-place replacement to quantized modules
mtq.quantize(model, quant_cfg, forward_loop=eval_func)
# model has FP8 qdq nodes at this point

input_img = torch.randn([1, 3, 640, 640])
with torch.no_grad():
    with export_torch_mode():
    # Compile the model with Torch-TensorRT Dynamo backend
        input_tensor = input_img.cuda()
        exp_program = torch.export.export(model, (input_tensor,))
        trt_model = torchtrt.dynamo.compile(
            exp_program,
            inputs=[input_tensor],
            enabled_precisions={torch.float8_e4m3fn},
            min_block_size=1,
            debug=False,
        )

        print(trt_model)`

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

narendasan commented 2 months ago

Seems like an issue with the dynamo export procedure. The main pytorch repo might be a better source of help.