Multiple images into one object

[pandov]

Hello! @shunsukesaito awesome work.

My question is the following: I have several images of the same person from different angles (the images were created at the same time), how can I interpolate one common object of better quality from these images? So that this object takes into account all projections. Is it possible to implement this feature using this solution?

I would be grateful if you can help me with this. Thanks!

[pandov]

I tried something and got a monster. I took a demo video result_ryota.mp4 and made new sample_images in different projections for my experiment. Visually looks like this:

Then in test.sh I changed the batch_size to 4 and additionally specified num_views to 4. I also made changes to the eval.py so that the image, mask and calib have 4 samples in the batch dimension (each image and mask are corresponding images from sample_images).

So, after all the changes, the result is the following. He seemed to have created all 4 models, connected them and removed all non-overlapping material (made a logical "and").

In my first post, I attached an image taken from the project page. I would like to get the same result. Send several images to the input and receive a 3D object composed of these images on the output.

[shunsukesaito]

Please refer to #3 for the previous discussions on this. As I said, the released model is not trained with multi-view setup. To support multi-view integration, you need not only modify the code as you did but also train a new model with this multiview setup (I'd recommend finetuning from the released model for faster convergence).

[pandov]

@shunsukesaito, I am not the only one who misunderstood. The complexity of the translation.

Then what is the idea behind the presented "Multi-View Reconstruction" approach? Just predicting multiple objects simultaneously from multiple input images?

[shunsukesaito]

Then what is the idea behind the presented "Multi-View Reconstruction" approach? Just predicting multiple objects simultaneously from multiple input images?

I think your initial understanding was correct. Given images from several view points, the network consolidates information from all the view points and generate the complete 3D geometry and texture as shown in the first image you attached.

[pandov]

@shunsukesaito, that's what I wanted to get. How can I combine given images from several view points to generate the complete 3D geometry and texture?

[shunsukesaito]

They are not currently supported in the released code but you could modify it for this purpose. You can refer to #45 #37 #31 #3 for reference.