Training without Instance Segmentation

[egeozsoy]

Is it possible to train the group-free network without having access to per point instance labels, so only using 3D bounding boxes? The loss calculation seems to depend on instance labels as far as I can tell

[stupidZZ]

Sorry, we did not fully understand your question, and our method does not rely on point instance labels. Can you provide more information on your concerns?

[egeozsoy]

Here https://github.com/zeliu98/Group-Free-3D/blob/ef8b7bb5c3bf5b49b957624595dc6a642b6d0036/scannet/scannet_detection_dataset.py#L74 you load the instance labels from scannet.

Here https://github.com/zeliu98/Group-Free-3D/blob/ef8b7bb5c3bf5b49b957624595dc6a642b6d0036/scannet/scannet_detection_dataset.py#L181 you use those instance labels to compute point_obj_mask and point_instance label.

And during the loss computation you use both the point_obj_mask https://github.com/zeliu98/Group-Free-3D/blob/ef8b7bb5c3bf5b49b957624595dc6a642b6d0036/models/loss_helper.py#L98 and the point_instance_label https://github.com/zeliu98/Group-Free-3D/blob/ef8b7bb5c3bf5b49b957624595dc6a642b6d0036/models/loss_helper.py#L25

How can the loss be computed without relying on the point instance labels?

[zeliu98]

Hi, this is used to assign a point to its' gt bbox.

• For scannet we follow votenet: https://github.com/facebookresearch/votenet/blob/2f6d6d36ff98d96901182e935afe48ccee82d566/scannet/load_scannet_data.py#L125 here they prepared the _ins_label.npy, and used in: https://github.com/facebookresearch/votenet/blob/2f6d6d36ff98d96901182e935afe48ccee82d566/scannet/scannet_detection_dataset.py#L131-L138 to calculate the point_votes and point_votes_mask, which have the similar use with us. Besides, they comment that

  # compute votes *AFTER* augmentation
  # generate votes
  # Note: since there's no map between bbox instance labels and
  # pc instance_labels (it had been filtered 
  # in the data preparation step) we'll compute the instance bbox
  # from the points sharing the same instance label. 

• For SUNRGBD, we assign the nearest gt bbox to a point if the point is inside it, please see https://github.com/zeliu98/Group-Free-3D/blob/ef8b7bb5c3bf5b49b957624595dc6a642b6d0036/sunrgbd/sunrgbd_data.py#L292-L326

Besides, you can refer to the preparation steps of SUNRGB for your need.