How to train Group-Free 3D model in SUN RGB-D dataset

[7erAcec0r9]

I was trying to train group-free 3d model in sun rgbd dataset and modified a config according to votenet and its scannet version config, but it seems has some problem with point dimension.

first I used the original model setting with backbone in_channel 3

type='PointNet2SASSG',
in_channels=3,

and I got the error:

RuntimeError: Given groups=1, weight of size [64, 3, 1, 1], expected input[1, 4, 2048, 64] to have 3 channels, but got 4 channels instead

then I changed the in_channel to 4 and got:

AssertionError: points dimension should be 3, but got unexpected shape 4

any ideas?

[JingweiZhang12]

Could you provide more logs to locate the line that reports an error?

[7erAcec0r9]

Could you provide more logs to locate the line that reports an error?

the backbone in_channels=3 report:

Traceback (most recent call last):
  File "/home/terace/mmdetection3d/tools/train.py", line 267, in <module>
    main()
  File "/home/terace/mmdetection3d/tools/train.py", line 256, in main
    train_model(
  File "/home/terace/mmdetection3d/mmdet3d/apis/train.py", line 346, in train_model
    train_detector(
  File "/home/terace/mmdetection3d/mmdet3d/apis/train.py", line 321, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
    return old_func(*args, **kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/detectors/base.py", line 60, in forward
    return self.forward_train(**kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/detectors/groupfree3dnet.py", line 54, in forward_train
    x = self.extract_feat(points_cat)
  File "/home/terace/mmdetection3d/mmdet3d/models/detectors/single_stage.py", line 61, in extract_feat
    x = self.backbone(points)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
    return old_func(*args, **kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/backbones/pointnet2_sa_ssg.py", line 118, in forward
    cur_xyz, cur_features, cur_indices = self.SA_modules[i](
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/ops/pointnet_modules/point_sa_module.py", line 210, in forward
    new_features = self.mlps[i](grouped_results)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/cnn/bricks/conv_module.py", line 207, in forward
    x = self.conv(x)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 442, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 1, 1], expected input[1, 4, 2048, 64] to have 3 channels, but got 4 channels instead

after changed into 4:

Traceback (most recent call last):
  File "/home/terace/mmdetection3d/tools/train.py", line 267, in <module>
    main()
  File "/home/terace/mmdetection3d/tools/train.py", line 256, in main
    train_model(
  File "/home/terace/mmdetection3d/mmdet3d/apis/train.py", line 346, in train_model
    train_detector(
  File "/home/terace/mmdetection3d/mmdet3d/apis/train.py", line 321, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
    return old_func(*args, **kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/detectors/base.py", line 60, in forward
    return self.forward_train(**kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/detectors/groupfree3dnet.py", line 58, in forward_train
    losses = self.bbox_head.loss(
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 205, in new_func
    return old_func(*args, **kwargs)
  File "/home/terace/mmdetection3d/mmdet3d/models/dense_heads/groupfree3d_head.py", line 424, in loss
    targets = self.get_targets(points, gt_bboxes_3d, gt_labels_3d,
  File "/home/terace/mmdetection3d/mmdet3d/models/dense_heads/groupfree3d_head.py", line 601, in get_targets
    objectness_masks) = multi_apply(self.get_targets_single, points,
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/core/utils/misc.py", line 30, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/terace/mmdetection3d/mmdet3d/models/dense_heads/groupfree3d_head.py", line 714, in get_targets_single
    box_indices_all = gt_bboxes_3d.points_in_boxes_part(points)
  File "/home/terace/mmdetection3d/mmdet3d/core/bbox/structures/base_box3d.py", line 532, in points_in_boxes_part
    box_idx = points_in_boxes_part(points,
  File "/home/terace/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/points_in_boxes.py", line 31, in points_in_boxes_part
    assert points.shape[2] == 3, \
AssertionError: points dimension should be 3, but got unexpected shape 4

are these enough?