Multi-view optimization

[guwa1]

Hey,

I was wondering whether it might be possible to improve the reconstruction performance by leveraging multiple images from different view angles, maybe in a similar fashion as it is done in One-2-3-45.

If you think it might be possible, I would be thankful for any pointer to the right direction.

Btw great work!

[Ayushlion8]

Hey @guwa1, try this:

1) Software Libraries: Consider using established open-source libraries and tools for multi-view stereo and 3D reconstruction. Some popular ones include:

2) COLMAP: A widely-used and powerful open-source software for SfM and MVS. It can be used for 3D reconstruction from images captured from different viewpoints. Meshroom: Another open-source tool for 3D reconstruction that is user-friendly and suitable for beginners. OpenMVS: A multi-view stereo reconstruction library that can generate dense 3D models from multiple images. Image Acquisition: Ensure that you have a set of images captured from different viewpoints of the object or scene you want to reconstruct. These images should ideally cover the object from all angles.

3) Calibration: If possible, calibrate your camera(s) to obtain intrinsic parameters such as focal length, principal point, and distortion coefficients. Accurate camera calibration is crucial for accurate reconstruction.

4) Feature Detection and Matching: Use feature detection and matching algorithms to identify common keypoints in the images. This step helps establish correspondences between images.

[ashawkey]

Thanks @Ayushlion8! @guwa1 Sorry that this repo only aims to perform single-view generation, you may check other methods for sparse view reconstruction, like this new paper today: https://hwjiang1510.github.io/LEAP/

[nagolinc]

@guwa1 Multiple-poses wouldn't actually be that hard to implement. You'd just have to imitate this block of code (https://github.com/dreamgaussian/dreamgaussian/blob/main/main.py#L174-L185 ) with the other images.

[nagolinc]

Okay, I tried loaded the multi-views generated by https://github.com/liuyuan-pal/SyncDreamer#syncdreamer into dream-gaussian.

Can't really tell if there's an improvement or not

https://github.com/dreamgaussian/dreamgaussian/assets/7775917/c058d864-b8c9-4233-836c-6b1eb470ee43

code, if you want to try it out

https://github.com/nagolinc/dreamgaussian/tree/mv

command-line should look like

python .\multi-cam.py --config .\configs\image-mv.yaml input=.\data\beaver_rgba.png save_path=beaver-mv imagesFile=.\data\beaver_mv.png camerasFile=.\data\camera-16.pkl

where beaver_mv is your output from syncdreamer

and camera-16 is

https://github.com/liuyuan-pal/SyncDreamer/blob/main/meta_info/camera-16.pkl

[leviathandragonflyer]

Would an mvdream compatible script be possible too?

[seakforzq]

Okay, I tried loaded the multi-views generated by https://github.com/liuyuan-pal/SyncDreamer#syncdreamer into dream-gaussian.

Can't really tell if there's an improvement or not

Blender.2023-10-04.19-35-45.mp4 code, if you want to try it out

https://github.com/nagolinc/dreamgaussian/tree/mv

command-line should look like

python .\multi-cam.py --config .\configs\image-mv.yaml input=.\data\beaver_rgba.png save_path=beaver-mv imagesFile=.\data\beaver_mv.png camerasFile=.\data\camera-16.pkl

where beaver_mv is your output from syncdreamer

and camera-16 is

https://github.com/liuyuan-pal/SyncDreamer/blob/main/meta_info/camera-16.pkl

do you try setting the weight of zero123 guidance as zero? maybe the two multi-views model are in conflicts so that the final result looks blurred?

[jclarkk]

@nagolinc I keep getting CUDA OOM when running your branch (both on V100 and A100):

Error running dreamgaussian: Traceback (most recent call last): File "~/Dev/dreamgaussian/multi-cam.py" , line 1241 , in <module> gui = GUI(opt) File "~/Dev/dreamgaussian/multi-cam.py" , line 98 , in __init__ self.renderer.initialize(num_pts=self.opt.num_pts) File "~/Dev/dreamgaussian/gs_renderer.py" , line 711 , in initialize self.gaussians.create_from_pcd(pcd, 10 ) File "~/Dev/dreamgaussian/gs_renderer.py" , line 345 , in create_from_pcd dist 2 = torch.clamp_min(distCUDA 2 (torch.from_numpy(np.asarray(pcd.points)).float().cuda()), 0.0000001 ) MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory Traceback (most recent call last): File "~/Dev/hexa-reconstruction/handlers/reconstruction.py" , line 133 , in handle mesh_file_path = mesh_handler.reconstruct_mesh_from_images( File "~/Dev/hexa-reconstruction/handlers/mesh/dreamgaussian.py" , line 98 , in reconstruct_mesh_from_images self.execute_command('dreamgaussian', dg_train_cmd, dg_path) File "~/Dev/hexa-reconstruction/handlers/mesh/abstract.py" , line 30 , in execute_command shell.execute_command(project, command, local_path, use_bash=use_bash) File "~/Dev/hexa-reconstruction/utils/shell.py" , line 29 , in execute_command raise Exception('Error running {}: {}'.format(project, parsed_output)) Exception: Error running dreamgaussian: Traceback (most recent call last): File "~/Dev/dreamgaussian/multi-cam.py" , line 1241 , in <module> gui = GUI(opt) File "~/Dev/dreamgaussian/multi-cam.py" , line 98 , in __init__ self.renderer.initialize(num_pts=self.opt.num_pts) File "~/Dev/dreamgaussian/gs_renderer.py" , line 711 , in initialize self.gaussians.create_from_pcd(pcd, 10 ) File "~/Dev/dreamgaussian/gs_renderer.py" , line 345 , in create_from_pcd dist 2 = torch.clamp_min(distCUDA 2 (torch.from_numpy(np.asarray(pcd.points)).float().cuda()), 0.0000001 ) MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

[jaggzh]

Thanks @Ayushlion8! @guwa1 Sorry that this repo only aims to perform single-view generation, you may check other methods for sparse view reconstruction, like this new paper today: https://hwjiang1510.github.io/LEAP/

They also point out:

The default training configurations require about 300GB at most, e.g. 8 A40 GPUs with 40GB VRAM, each.

[ptyogi]

hello. In this video, I used github here as a reference to create a nice model using two images. https://youtu.be/1xv3NBIYT44?feature=shared&t=212 It seems that the model above is created using 16 images, but how can I create it using 2 images?