Config issue

[YoushaaMurhij]

Thanks for sharing this great work! I tried to run inference using raydn_r50_704_bs2_seq_428q_nui_60e.py but got this error:

    outputs = self.model(return_loss=False, rescale=True, **data)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/mmcv/parallel/data_parallel.py", line 51, in forward
    return super().forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/fp16_utils.py", line 208, in new_func
    return old_func(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/detectors/repdetr3d.py", line 218, in forward
    return self.forward_test(**data)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/detectors/repdetr3d.py", line 288, in forward_test
    return self.simple_test(img_metas[0], **data)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/detectors/repdetr3d.py", line 315, in simple_test
    bbox_pts = self.simple_test_pts(
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/detectors/repdetr3d.py", line 301, in simple_test_pts
    outs = self.pts_bbox_head(img_metas, **data)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/dense_heads/raydn_head.py", line 524, in forward
    outs_dec = self.transformer(tgt, query_pos, feat_flatten, spatial_flatten, level_start_index, temp_memory,
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/utils/detr3d_transformer.py", line 110, in forward
    inter_states = self.decoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/utils/detr3d_transformer.py", line 171, in forward
    query = layer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/utils/detr3d_transformer.py", line 460, in forward
    x = self._forward(
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/utils/detr3d_transformer.py", line 399, in _forward
    query = self.attentions[attn_index](
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/RayDN/projects/mmdet3d_plugin/models/utils/detr3d_transformer.py", line 527, in forward
    output = self.output_proj(features)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (428x128 and 256x256)

Could you please guide me to fix it? I also wonder if I need to change anything in StreamPETR baseline in case I want to merge RayDN head into it and use this config?

[LiewFeng]

Hi,@YoushaaMurhij, thank you for your interest in our work. It looks like that the fature dim is wrong. It shoud be 256 as shown in the config and config. It may due to that you modify the hyper-parameters.

[YoushaaMurhij]

Thanks for your response!. I forgot to set the correct number of cameras.

[YoushaaMurhij]

@LiewFeng, I am trying to inference raydn_r50_704_bs2_seq_428q_nui_60e on my own data. I set num_cams = 3 in DeformableFeatureAggregationCuda as I have only 3 cameras. I visualized the results and there were only boxes from the front camera. Do I need to modify addtional model parameters to get it work on 3 cameras? Using StreamPETR r50 config gives results from all 3 cameras. Any suggestion would be appreciated! Thank you!