search

monniert / differentiable-blocksworld

[NeurIPS 2023] Code for "Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives"
https://www.tmonnier.com/DBW
MIT License
197 stars 10 forks source link

Train on custom scene #12

Open MartinGGS opened 1 month ago

MartinGGS commented 1 month ago

I hope you are doing well. I wanted to start by thanking you for the excellent work and for making such a complete implementation available. I have been working with your code and it has been very useful for my research.

I am currently trying to train the model with custom scenes using Nerfstudio to prepare the cameras and data, but I had some questions regarding how to adjust the R_world and T_world values ​​in the settings. I would like to know more details on how to estimate these values. I have read in the documentation that visual comparisons can be done with tools like Plotly or Blender, but could you provide more information on the tuning process? Are there any additional tips or recommendations to get a good starting point for this?

Also, I am interested in better understanding how pseudo-ground truth is handled in custom scenes with these values.

I really appreciate your time and any additional details you can share on this topic.

Thanks again for all the work done!

monniert commented 1 week ago

Hey @MartinGGS, thanks for the message and sorry for the delay!

In general, I use a notebook where I instantiate a blockworld model with identity R_wold and T_world. Then, for the target scene, I load the associated point cloud (which typically comes as an output of your MVS algorithm, either COLMAP or Nerfstudio). Finally, using Plotly, I plot the point cloud along with the meshes of the BW model and coarsely modify R_wold and T_world such that there is a global alignment between the grounds.

Hope that helps!