Point cloud RGB and ground truth sometimes can be misleading

[TerryLiu007]

I find most of the RGB information come with the point cloud is unusable. I understand it's a issue with point sampling from mesh. Another related problem is the ground truth labeling - when two mesh surfaces are very close to each other and belong to different class, the sampled points are almost on the same surface. This kind of labeled data would be harmful to train the network I suppose? Could such data be taken out for the future release?

Some examples are shown below, like the inside and outside of a lamp base, and the bed sheet and bed board. The left model is shown with RGB info and right model is shown with the ground truth label.

[daerduoCarey]

Hi, @TerryLiu007,

Thank you for a lot for your interests in PartNet. We are making it better and better. Thank you for the patient.

1) For the RGB issue, the point colors are directly sampled from the ShapeNet models. As you observed, sometimes there are two kinds of colors that are overlapping. This is because most ShapeNet models have two surfaces: one outer surface with the correct colors/textures/materials, one inner surface that is mostly gray/white/black. There is no easy way to tell which one is inner and which one is outer for now, as there may be holes/inside parts on the meshes. We are trying to get an outer-surface-only version by sampling from outside camera views and only use the points that we can see from any of the views. It produces a much better result that removes the overlapping colors, but it fails to preserve the inside parts. So, we welcome any idea on this from the community!

2) Regarding the overlapping part labels, this is mostly due to the data annotation error, as the lamp example. You can try to use this link (and change 2230 to the anno_id you are looking for) to check if the annotation has overlapping parts. If so, this is probably an annotation error and you are warmly welcomed to let us know by filling in this errata. We are also actively working on improving the data quality. We have done two-round of verifications for releasing this PartNet v0. We believe the error rate should be <1% if you count by parts and <5% if you count by shapes. We will fix problematic annotations as much as we can for the next release.

Thank you Kaichun

[heesooy]

1. For the RGB issue, the point colors are directly sampled from the ShapeNet models. As you observed, sometimes there are two kinds of colors that are overlapping. This is because most ShapeNet models have two surfaces: one outer surface with the correct colors/textures/materials, one inner surface that is mostly gray/white/black. There is no easy way to tell which one is inner and which one is outer for now, as there may be holes/inside parts on the meshes. We are trying to get an outer-surface-only version by sampling from outside camera views and only use the points that we can see from any of the views. It produces a much better result that removes the overlapping colors, but it fails to preserve the inside parts. So, we welcome any idea on this from the community!

HI @daerduoCarey Out of curiosity, are there any updates on this outer-surface-only version? For my use case, the inside parts being lost is okay. I am planning on using the objects in the context of a larger scene (i.e. placing objects into a room), in which occlusion would be expected. Thanks!