omv-ysong
commented
2 years ago Hi, I spent quit a while checking this issue. This tool aims to converting rendered depth and color images to underwater for robotic simulation platforms, when utilizing real world depth maps, the imperfect depth maps will cause several artifacts which require additional pre-processing.
- about Red: I fixed this bug by checking "cos_theta" in rendering.cpp, and avoid negative values which indicate no light directly hit to this point.
- about Blue: this is still a open question. Middlebury dataset uses structured light to capture ground truth depth, the stereo shadow between the camera and the infrared emitter will introduce missing data and contour artifacts. So the input imperfect depth map tells that there is a 2mm depth scratch on the surface, the renderer will render a darker shadow in this area accordingly (especially with point light sources). This is a depth map refinement issue and is out of the scope of this renderer. If you want to use Middlebury depth maps as GT, I would suggest you to keep this effect as it reflects the "true" geometry information.
- about Green, I add the ambient light component in the renderer, you can adjust the factor (light_ambient_RGB) in the yaml config file.
The attached files you uploaded somehow have different image sizes with the original middlebury data, i uploaded the tool in another repo: https://github.com/omv-ysong/DSV/blob/93f79302607fd10f5d0a77cf768881cf1545cf6c/tools/convert_disparity_to_depth.py You can use it to generate .exr depth map. Since it still contains empty depth value, i add an depth inpainting function in the renderer, you can turn on it in the yaml config file (refine_depth: true).
This is the test result:
using following config parameters: scale_factor: 4.0 scale_factor_bs: 8000.0 render_back_scatter: true volumetric_max_depth: 8.0 num_volumetric_slabs: 8 write_slab: false slab_sampling_method: 1 image_width_height: [2632, 1988] camera_intrinsic_matrix: [7315.238, 0.0, 997.555, 0.0, 7315.238, 980.754, 0.0, 0.0, 1.0] white_balance: [2.28613, 1.0, 1.361328] water_attenuation_RGB: [0.37, 0.044, 0.035] vsf_type: 3 light_spectrum_RGB: [0.25, 0.35, 0.4] light_ambient_RGB: [0.4, 0.4, 0.4] light_RID_type: 1 num_lights: 1 light_positions_XYZ: [[0.5, 0.0, 0.0]] light_orientations_RollPitchYaw: [[0.0, 0.0, 0.0]] write_uw_img_in_exr: false depth_sommoth_window_size: 5 refine_depth: true auto_iso: false
Background: I am trying to generate images for evaluation of depth-from-stereo algorithms for a deep sea environment. Since there are none I am aware of, I'm trying to make deep sea versions of the Middlebury dataset (and MPI Sintel).
I am encountering 2-3 types of Rendering bugs in the DeepSeaRenderer, as shown in the attached picture (based on "Jadeplant" from the 2014 Middlebury dataset).
Red: happening in ConvertDoubleMatTo8Bit() due to negative values not beeing checked for and causing an underflow.
Blue: "Shadows" caused by small differences in depth of neighboring pixels (> ~0.01). Does look unnatural..
Green: Rough edges. caused by bigger differences in depth of neighboring pixels (> ~0.1). Does look very unnatural..
Red can be fixed by adding the following lines to ConvertDoubleMatTo8Bit():
I also noticed that the auto_iso setting is ignored in 2 of 3 versions of this function.
Blue and Green may have the same cause, but I don't know how to improve it. I tried blurring the depth map, which mostly takes care of Blue, but makes Green look even worse. I noticed these bugs seem not to happen in your testdata. Do you have any ideas why?
I tried various configuration parameters, starting with the config file from the testdata and several pictures from the Middlebury and MPI Sintel datasets and had these bugs everywhere (with one exception that did not have Red).
I tried attaching the image and my computed depth map (as exr (hope that saved right..) and as .pfm -- which is what Middlebury uses.), but as that didn't work, I uploaded it here: https://aguzinski.de/BWPubq6/Jadeplant.zip