search

noahstier / vortx

Source code for the paper "Volumetric 3D Reconstruction with Transformers for Voxel-wise View Selection and Fusion"
MIT License
67 stars 10 forks source link

Parameters for inference on ICL-NUIM and TUM RGB-D data #7

Open merlushka opened 1 year ago

merlushka commented 1 year ago

Hello! Thank you for you work and code! Could you please explain the details of inference on ICL-NUIM and TUM RGB-D datasets that you described in your paper? I try to inference the ICL-NUIM scenes using the same checkpoint and the same parameters as written in the config for ScanNet test, but the reconstructions turn out be much worse and I do not get the metrics you got.

Thank you in advance

noahstier commented 1 year ago

Hi, the details for running inference on those datasets are the same as for ScanNet. As a baseline, are you able to reproduce the inference results on ScanNet?

If you are having trouble reproducing the TUM RGB-D and ICL-NUIM results from the paper my first guess is that you might not be using the correct pose or camera intrinsics. To confirm, are you able to generate coherent point cloud or TSDF reconstructions for those datasets using the ground-truth depth?

merlushka commented 1 year ago

Thank you! Yes, the inference on ScanNet works well. Probably something really went wrong with camera parameters. Should I use the initial poses and intrinsics (with negative focal length) from ICL-NUIM without any reflection or rotation?

noahstier commented 1 year ago

The trained VoRTX model is expecting the scene's gravitational axis to be aligned with the world Z axis but it the ICL-NUIM poses are aligned to the Y axis instead so you will need to swap the axes. Just make sure you can get a good reconstruction using the ground-truth depth images, with the scene's up axis aligned with world Z, then you know you have the correct pose & intrinsics for VoRTX.

merlushka commented 1 year ago

Ok, got it! A have swaped the axis so that z-axis is vertical. Now the reconstruction looks quite good, but I still can not evaluate the quality. When I try to render the depth from the mesh, I get wrong depth maps due to negative focal length (fy). I can use the absolute value of focal length and flip the rendered depth. Or have you done something else to get correct depth maps?

noahstier commented 1 year ago

the reconstruction looks quite good

Great!

I don't believe we ever rendered depths from those reconstructions so I never ran into the issue you're describing. Actually I don't really understand why the negative focal length is a problem. Flipping the depth sounds like a good workaround though. You might just want to verify the flipped depth by back-projecting to make sure the points fall exactly on the mesh.

merlushka commented 1 year ago

Ok, thank you! I actually don't need depths, but wanted to run evaluation. And your implementation of metrics computing required depth rendering (I used scripts/evaluate.py). Did you evaluate the quality of reconstruction on ICL-NUIM differently?

merlushka commented 1 year ago

If you have the code for ICL-NUIM processing, and reproducing and evaluation the results on this datasets, could you please share it?

I tried to inverse the negative focal length and rotate so that z-axis is vertical. The points fall exactly to the mesh after backprojecting and the reconstruction seems quite good. However, I do not get the same evaluation results as you mentioned in the paper.

SwingWillwow commented 1 year ago

Ok, thank you! I actually don't need depths, but wanted to run evaluation. And your implementation of metrics computing required depth rendering (I used scripts/evaluate.py). Did you evaluate the quality of reconstruction on ICL-NUIM differently?

Hi! I face the same problem that the reconstruction quality is good but failed to evaluate since the rendered depth is bad. Have you solved this problem now?

passingdragon commented 4 months ago

If you have the code for ICL-NUIM processing, and reproducing and evaluation the results on this datasets, could you please share it?

I tried to inverse the negative focal length and rotate so that z-axis is vertical. The points fall exactly to the mesh after backprojecting and the reconstruction seems quite good. However, I do not get the same evaluation results as you mentioned in the paper.

Can you give me the preprocessor or the processed data about TUM-RGBD and ICL-NUIM datasets for evaluation? Please

passingdragon commented 4 months ago

Ok, thank you! I actually don't need depths, but wanted to run evaluation. And your implementation of metrics computing required depth rendering (I used scripts/evaluate.py). Did you evaluate the quality of reconstruction on ICL-NUIM differently?

Hi! I face the same problem that the reconstruction quality is good but failed to evaluate since the rendered depth is bad. Have you solved this problem now?

Can you solve this problem?I encountered some difficulties in data processing during my undergraduate graduation. Could you please provide me with the pre-processing program or the data after processing about TUM-RGBD or ICL-NUIM in the paper? Thank you

passingdragon commented 4 months ago

If you have the code for ICL-NUIM processing, and reproducing and evaluation the results on this datasets, could you please share it?

I tried to inverse the negative focal length and rotate so that z-axis is vertical. The points fall exactly to the mesh after backprojecting and the reconstruction seems quite good. However, I do not get the same evaluation results as you mentioned in the paper.

Can you solve this problem?I encountered some difficulties in data processing during my undergraduate graduation. Could you please provide me with the pre-processing program or the data after processing about TUM-RGBD or ICL-NUIM in the paper? Thank you