search

alexsax / 2D-3D-Semantics

The data skeleton from Joint 2D-3D-Semantic Data for Indoor Scene Understanding
http://3dsemantics.stanford.edu
Apache License 2.0
464 stars 67 forks source link

Converting normal image to surface normals #13

Closed meder411 closed 6 years ago

meder411 commented 6 years ago

You mention that the same convention is used as in NYU Depth, but the paper for NYU Depth v2 doesn't lay out their surface normals in detail either nor does their toolbox include an example of extracting normals from the image. What is the proper way to convert the RGB values to normals? I imagine the first step is to subtract 127.5, but after that I'm not quite sure.

From the paper:

The surface normals in 3D corresponding to each pixel are computed from the 3D mesh instead of directly from the depth image. The normal vector is saved in the RGB color value where Red is the horizontal value (more red to the right), Green is vertical (more green downwards), and Blue is towards the camera. Each channel is 127.5-centered, so both values to the left and right (of the axis) are possible. For example, a surface normal pointing straight at the camera would be colored (128, 128, 255) since pixels must be integer-valued.

Does this mean that I do n = (img - 127.5) / 127.5? If so, what do we do about quantization errors due to the integer casting? Also, this doesn't appear to be a right-handed coordinate system.

alexsax commented 6 years ago

Yes, that's the right way to recover surface normals from the RGB encoding.

We chose to follow the coloring in the NYU depth dataset so that those familiar with one dataset can easily parse the other. Of course, you can get a right-handed coordinate system by flipping one of the axes :)

Regarding quantization noise: With the current state of scanning technology the reconstructed surfaces are simpler than the true surfaces. In any scanned mesh, this approximation error will be much greater than the quantization error. In addition, current normal estimation techniques usually have errors that are significantly larger than the noise introduced by quantization. However, if you really can't accept any additional noise in the labels then you could do some rule-based smoothing (or something more sophisticated like a CRF).

meder411 commented 6 years ago

Okay, thanks a lot for clearing this up! Once I'm done with my work on this dataset, perhaps I'll submit my Python dataloader as a PR to help out future users.

alexsax commented 6 years ago

That would be very welcome!