search

THU-DA-6D-Pose-Group / GDR-Net

GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation. (CVPR 2021)
https://github.com/THU-DA-6D-Pose-Group/GDR-Net
Apache License 2.0
269 stars 46 forks source link

Missing annotated data when training with real+syn LMO dataset #60

Closed GabrielleTse closed 1 year ago

GabrielleTse commented 2 years ago

Hi Dr. Wang,

Me again.

We try to train with the LM-O real+syn data (as shown below). image

However, we find that the syn data (which should be placed in the folder datasets/lm_renders_blender) generated by pvnet-rendering can not be used directly. As show in the training configuration file GDR-Net/core/gdrn_modeling/datasets/lm_blender.py, three annotated data are missing, which are obj_gt.json, scene_mask_opengl.png, scene_xyz_bop.pkl, respectively. image image Could you please provide the data structure of those files, or is there any reference code we can have for accomplishing the experiment reproduce?

wangg12 commented 2 years ago

The json is just a collection of annotations of each image generated by pvnet-rendering. mask_opengl can be obtained by rendering the pose or just use the mask genereted by pvnet-rendering. The xyz can also be generated by rendering the 3D model with the poses, just adapt this reference code https://github.com/THU-DA-6D-Pose-Group/GDR-Net/blob/main/tools/lm/lm_pbr_1_gen_xyz_crop.py.

BTW, we strongly suggest you to use the publically available pbr synthetic data instead of this pvnet-rendering data for reproducibility, because no one can ensure the data generated by pvnet-rendering is the same.

GabrielleTse commented 2 years ago

Agree. We'd like to use the publicly available data to train LM-O.

For pbr synthetic data, part of them can be downloaded from BOP website, and part of them can be generated by the provided generation tool code. This is used to verify the result of real+pbr, as shown below. image

For the result of real+syn, as shown below, there is neither publicly available 10k synthetic images syn, nor supporting files for training. image

According to the training configuration for LM-O dataset, GDR-Net/configs/gdrn/lmo/a6_cPnP_AugAAETrunc_BG0.5_lmo_real_blender_160e.py, lm_renders_blender is necessary for training real+syn. That's why we have to generate the syn data through the pvnet-rendering link. image

To train the pvnet-rendering generated data, supporting files obj_gt.json, scene_mask_opengl.png, scene_xyz_bop.pkl are needed, but we can not find the code to generate those files yet, that's why we ask for help.

To this end, could you please kindly provide the pvnet-rendering generated data and the supporting training files, or could you please provide the code to generate obj_gt.json, scene_mask_opengl.png, scene_xyz_bop.pkl? Thank you very much for your attention and assistance!

wangg12 commented 2 years ago

Please kindly refer to

The json is just a collection of annotations of each image generated by pvnet-rendering. mask_opengl can be obtained by rendering the pose or just use the mask genereted by pvnet-rendering. The xyz can also be generated by rendering the 3D model with the poses, just adapt this reference code https://github.com/THU-DA-6D-Pose-Group/GDR-Net/blob/main/tools/lm/lm_pbr_1_gen_xyz_crop.py.

for generating obj_gt.json, scene_mask_opengl.png, scene_xyz_bop.pkl.