alicevision / Meshroom

3D Reconstruction Software
http://alicevision.org
Other
11.19k stars 1.08k forks source link

[request] Add possibility to import depth information/estimation from an external source as a pipeline node #2534

Open imDema opened 1 month ago

imDema commented 1 month ago

Problem description

I am having problems reconstructing a scene taken with a not so good camera. The problem I would like to address is the quality of the depth estimation performed by the DepthMap node.

While trying to tune my pipeline to get a better result, I got to inspecting the depth estimation produced by the depth map node. To my surprise I found the depth estimation to be extremely noisy, to the point that it got me wondering if it were the cause of the bad reconstruction (image below)

Proposed solution

Having experience with neural network methods for depth estimation, I tried computing the same depth estimate using MiDaS and, as expected I got an incredibly better result (image below).

While I do understand that including the MiDaS or another ML based approach for depth estimation in Meshroom could lead to a set of problems and may not be desirable, I think that adding an option to outsource the depth estimation step to an external tool for advanced users could be very beneficial to the quality of the output. This would also come in handy in cases when depth information for the image is already available thanks to ToF sensors or other, which is also a concern that has been raised in other issues.

Alternatives

Additional context This issue is related to #1493, however I could not find an actual implemented/documented solution to do it even though it is marked as closed.


Here are some comparison between the meshroom output and the MiDaS output run on the same photos (i resized and reencoded them to jpeg manually for this issue)

Sample 1

Meshroom: 51780469_depthMap_Meshroom MiDaS: 51780469_depthMap_MiDaS

Sample 2

Meshroom: 765218757_depthMap_Meshroom MiDaS: 765218757_depthMap_MiDaS

Sample 3

Meshroom: 1940366396_depthMap_Meshroom MiDaS: 1940366396_depthMap_MiDaS

EDIT: this is the script I used to compute the depth maps

#!/bin/bash

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters"
fi

for file in $1/*.exr; do
        filename=$(basename "$file" .exr)
        magick "$file" -resize 50% -filter Lanczos "input/${filename}.jpg"
done
python3 run.py --model_type dpt_beit_large_512 --input input/ --output output/ --grayscale

SUFFIX=-dpt_beit_large_512.png

for file in output/*${SUFFIX}; do
        filename=$(basename "$file" ${SUFFIX})
        magick "$file" -compress Lossless "exr/${filename}_depthMap.exr"
done
natowi commented 3 days ago

You may want to take a look at https://github.com/alicevision/MeshroomResearch and https://github.com/alicevision/MeshroomResearch/blob/main/mrrs/nodes/utils/DepthMapTransform.py for example

you can build a custom python node. The next release will support easier addin/custom node creation

A script was also shared by remmel, more discussion here https://github.com/alicevision/meshroom/issues/1361