search

open-mmlab / mmaction2

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark
https://mmaction2.readthedocs.io
Apache License 2.0
4.23k stars 1.24k forks source link

RuntimeError: Expected 5-dimensional input for 5-dimensional weight [64, 3, 3, 7, 7], but got 4-dimensional input of size [64, 1, 256, 256] instead #429

Closed wwdok closed 3 years ago

wwdok commented 3 years ago

Hi, there, i have succefully converted TSN .pth model to .onnx format in colab, and now i want to convert other models like ircsnto to onnx format, but i came across an error as the title described. The command i run in colab : !python tools/pytorch2onnx.py ./configs/recognition/csn/ircsn_ig65m_pretrained_bnfrozen_r152_32x2x1_58e_kinetics400_rgb.py ./checkpoints/ircsn_ig65m_pretrained_bnfrozen_r152_32x2x1_58e_kinetics400_rgb_20200812-9037a758.pth --shape 32 2 1 256 256 --verify --show Its output logs are : 

Traceback (most recent call last):
  File "tools/pytorch2onnx.py", line 163, in <module>
    verify=args.verify)
  File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
    opset_version=opset_version)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
    outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 59, in forward_dummy
    x = self.extract_feat(imgs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/base.py", line 72, in extract_feat
    x = self.backbone(imgs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/backbones/resnet3d.py", line 795, in forward
    x = self.conv1(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/cnn/bricks/conv_module.py", line 192, in forward
    x = self.conv(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/cnn/bricks/wrappers.py", line 79, in forward
    return super().forward(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 485, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [64, 3, 3, 7, 7], but got 4-dimensional input of size [64, 1, 256, 256] instead

I also tried on i3d and tpn, they all have similiar errors, what is the common reason for this type of error ? Thanks !

SuX97 commented 3 years ago

The shape is not correct. Check this doc out:

--shape: The shape of input tensor to the model. For 2D recognizer(e.g. TSN), the input should be $batch $clip $channel $height $width(e.g. 1 1 3 224 224); For 3D recognizer(e.g. I3D), the input should be $batch $clip $channel $time $height $width(e.g. 1 1 3 32 224 224); For localizer such as BSN, the input for each module is different, please check the forward function for it. If not specified, it will be set to 1 1 3 224 224.

wwdok commented 3 years ago

Ah, I overlooked this point ! The extra parameter time means sampling frame interval ？ I have generated following onnx files, i find the $time $height $width do not affect file size, is this reasonable ？ image By the way, should $batch $clip $channel always be 1 1 3 ?

But this input tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :

Traceback (most recent call last):
  File "tools/pytorch2onnx.py", line 163, in <module>
    verify=args.verify)
  File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
    opset_version=opset_version)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
    outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 60, in forward_dummy
    outs = (self.cls_head(x), )
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/heads/tpn_head.py", line 48, in forward
    kernel_size = (1, x.shape[-2], x.shape[-1])
AttributeError: 'tuple' object has no attribute 'shape'
dreamerlin commented 3 years ago

Ah, I overlooked this point ! The extra parameter time means sampling frame interval ？ I have generated following onnx files, i find the $time $height $width do not affect file size, is this reasonable ？ image By the way, should $batch $clip $channel always be 1 1 3 ?

But this input tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :

Traceback (most recent call last):
  File "tools/pytorch2onnx.py", line 163, in <module>
    verify=args.verify)
  File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
    opset_version=opset_version)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
    outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 60, in forward_dummy
    outs = (self.cls_head(x), )
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/heads/tpn_head.py", line 48, in forward
    kernel_size = (1, x.shape[-2], x.shape[-1])
AttributeError: 'tuple' object has no attribute 'shape'

It seems the neck is somehow skipped

SuX97 commented 3 years ago

It seems the neck is somehow skipped

@dreamerlin True, forward_dummy is used as the inference code. There is no neck in it.

SuX97 commented 3 years ago

Ah, I overlooked this point ! The extra parameter time means sampling frame interval ？ I have generated following onnx files, i find the $time $height $width do not affect file size, is this reasonable ？ image By the way, should $batch $clip $channel always be 1 1 3 ?

  1. time means the clip_length. 2. The size is determined mostly by the number of parameters. Therefore it is not input-sensitive. 3. 1 1 3 would work.

But this input tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :

There is a bug for TPN conversion, which uses a neck module. We will fix it.