search

hkust-vgd / scenenn

Supplemental code and scripts for the paper SceneNN: A Scene Meshes Dataset with aNNotations
http://www.scenenn.net
73 stars 18 forks source link

Misalignment of 2D annotations #5

Open orestis-z opened 6 years ago

orestis-z commented 6 years ago

I want to label the raw RGBD data with 2D annotations. The depth and RGB data align well. Whereas the annotations do not seem to align with the raw RGBD data.

Take a look at the ladder on the right: scenenn

Could you point me to how I need to transform the annotations (or rgbd) in order to align well?

orestis-z commented 6 years ago

Looks like the labeling is done on the depth but not transformed onto RGB using extrinsics: image Depth (darker region) on RGB: image The images are from 213/image/image03404.png

@songuke could you please comment on that? Thank you.

songuke commented 6 years ago

@zamponotiropita The annotation was done on the 3D mesh reconstructed from the RGB-D video, then reproject to 2D. Most of the time this works, but in some scenes the reconstruction is not perfect (some misalignments during 3D reconstruction), and this causes misalignments in reprojection to 2D too.

We attempted to address this issue by doing some optimizations to realign the boundaries (see here: http://people.sutd.edu.sg/~saikit/projects/sceneNN/pdf/annotation_tvcg17.pdf), but obviously this is not suitable if you want to address every frame in the scene. I think you might want to pick a scene with good 2D alignment if this really affects your application.

(Somehow I was not notified when issues are posted on Github, hence this late response. I will check my Github again. Sorry for that.)

KengChiLiu commented 6 years ago

@songuke In 2D annotation link you provide, it is empty in depth and rgb folder ??? Does scenenn provide ready both aligned RGBD and semantic annotation in 2D ???

thank you

songuke commented 6 years ago

Yes, the depth and RGB data are provided in the compressed RGB-D video in the ONI files. If you need it, please use the provided playback tool to export.

For semantic annotation, we provide the annotated data in 3D mesh. If you need 2D information, we provided the annotation tool that can be used to reproject all the annotation to 2D. The 2D annotation will be aligned to the color image in most cases. Only in a few rare scenes, the 3D reconstruction is not quite good, and so the 2D-3D alignment might be off a little bit.

To obtain exact alignment, one way we thought about is to reproject both depth, color, and annotation using the camera poses. The disadvantage of this approach is that the reprojected color can be slightly blurry compared to raw color image (since the reprojected color image is from the 3D mesh textures).

2018-04-25 12:11 GMT+09:00 Keng-Chi Liu notifications@github.com:

@songuke https://github.com/songuke In 2D annotation link you provide, it is empty in depth and rgb folder ??? Does scenenn provide ready both aligned RGBD and semantic annotation in 2D ???

thank you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scenenn/scenenn/issues/5#issuecomment-384147723, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIXsrHloVX4ka8ggOJPqayAOW1D4Rbqks5tr-lqgaJpZM4QrmPU .