Need Clarification - Generating symmetries.txt With symseg

[cm107]

According to Dataset set-up section in README.md, symmetry labels were generated using symseg. It is not clearly stated in README.md (😭), but specifically what needs to be done is the generation of symmetries.txt.

If you follow the setup procedure in README.md, these symmetries.txt files are downloaded automatically when you execute:

python data/download_linemod.py

and can be found at data/linemod/symmetries/*/symmetries.txt.

Since I'm making my own dataset with my own classes, I need to generate data/linemod/keypoints/*/keypoints_3d.npy and data/linemod/symmetries/*/symmetries.txt for each of my classes. I was able to generate keypoints_3d.npy using the method shown below:

Show keypoints_3d.npy generation method

Execute the following script in [clean-pvnet](https://github.com/KatharinaSchmidt/clean-pvnet)'s root directory: ```python= from argparse import ArgumentParser from lib.csrc.fps.fps_utils import farthest_point_sampling from lib.utils.linemod.opengl_renderer import OpenGLRenderer import numpy as np def parse_args(): parser = ArgumentParser() # training parser.add_argument('--ply_path', type=str, required=True) parser.add_argument('--save_path', type=str, default='model_kpts3d.npy') parser.add_argument('--num_keypoints', type=int, default=8) args = parser.parse_args() return args args = parse_args() renderer = OpenGLRenderer(args.ply_path) sampled_kpts3d = farthest_point_sampling(pts=renderer.model['pts']/1000, sn=args.num_keypoints, init_center=True) np.save(args.save_path, sampled_kpts3d) ```

Next I need to generate symmetries.txt, so I setup symseg and confirmed that the example provided is working correctly. However, I have no idea what to do from here. I was expecting to see some sort of API with options for creating a symmetries.txt dump file, but I can't seem to find that anywhere. (Is there even an API for this tool at all?)

I need some clarification on how specifically the symmetries.txt files were created using symseg. Also, if no symmetries are found, does that mean it is impossible to correctly create labels for training the HybridPose model?

[chensong1995]

Hello cm107,

Thanks for your interest in our work!

1. SymSeg will output a text file to symseg/bin/output. It has the same format as symmetries.txt used in HybridPose.
2. SymSeg has quite a few tunable hyper-parameters. It should always be possible to detect the symmetry plane with the right combination of hyper-parameters. In the case of SymSeg failure, you can also label the 3D symmetry plane manually, and create a symmetries.txt file according to your manual label.
3. After generating symmetries.txt, you will need to create 2D labels. This can be down by writing a short script based on the ones we provided here.

I hope this helps.

[cm107]

@chensong1995 Thanks for the quick reply!

SymSeg will output a text file to symseg/bin/output. It has the same format as symmetries.txt used in HybridPose.

Thanks to your hint, I was able to find the relevant code in examples/reflectional_detection/main.cpp. Using the following command I was able to successfully generate symmetries.txt for sample_object_cloud.ply:

cd symseg/bin
mkdir output
./reflectional_detection ../sample_object_cloud.ply -novis output

output:

clayton@asus:~/workspace/git/symseg/bin$ ./reflectional_detection ../sample_object_cloud.ply -novis output
Loading data...
[pcl::PLYReader] ../sample_object_cloud.ply:28: property 'float32 focal' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:29: property 'float32 scalex' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:30: property 'float32 scaley' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:31: property 'float32 centerx' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:32: property 'float32 centery' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:35: property 'float32 k1' of element 'camera' is not handled
[pcl::PLYReader] ../sample_object_cloud.ply:36: property 'float32 k2' of element 'camera' is not handled
Downsampling input pointcloud...
11632 points in original cloud.
7418 points after downsampling.
Detecting reflectional symmetry...
  1 symmetries detected.
  2.88307 seconds.
----------------------------
Total time: 2.88469 seconds
Saving segmentation results to output/symmetries.txt

Unfortunately I am not detecting any symmetries for the model that I want to use.

clayton@asus:~/workspace/git/symseg/bin$ ./reflectional_detection /home/clayton/workspace/prj/data/misc_dataset/custom_ref/model.ply -novis output
Loading data...
[pcl::PLYReader] /home/clayton/workspace/prj/data/misc_dataset/custom_ref/model.ply:14: property 'list uint8 uint32 vertex_indices' of element 'face' is not handled
Failed to find match for field 'rgb'.
Failed to find match for field 'curvature'.
Downsampling input pointcloud...
3282581 points in original cloud.
8500 points after downsampling.
Detecting reflectional symmetry...
  0 symmetries detected.
  2.1852 seconds.
----------------------------
Total time: 2.33122 seconds
Saving segmentation results to output/symmetries.txt

I'm not quite sure what the cause is, but maybe there are too many points in the original cloud before downsampling.

SymSeg has quite a few tunable hyper-parameters. It should always be possible to detect the symmetry plane with the right combination of hyper-parameters. In the case of SymSeg failure, you can also label the 3D symmetry plane manually, and create a symmetries.txt file according to your manual label.

There seem to be many hyperparameters defined in examples/reflectional_detection/config.pbtxt. Are there any parameters in particular that would help me solve the above problem?

[chensong1995]

Hi cm107,

It has been over a year since I generated the symmetry labels, so I can't exactly recall what I did with the hyper-parameters in SymSeg. Here are some tips:

1. If I remember correctly, SymSeg requires the input to have per vertex normal information. If this is unavailable in your data, you can use meshlab to estimate the normals and write them in the object file.
2. It does take some patience to adjust the hyper-parameters in SymSeg. I remember setting a few breakpoints in their source code, and use gdb to investigate what each part does and which part goes wrong.
3. As a last resort, you can manually label the 3D symmetry plane. In meshlab, you can translate the object center to the origin, and rotate so the symmetry plane aligns with xOy. This gives a trival point-normal representation of the symmetry plane.

I hope this helps.

[cm107]

Thanks for the tips! I was able to generate the symmetries.txt after modifying the .ply file to have a more uniform external mesh like the linemod models. I also removed internal parts so that only the external mesh remained.