search

Lakonik / MVEdit

[WIP] Generic 3D Diffusion Adapter Using Controlled Multi-View Editing
https://lakonik.github.io/mvedit/
MIT License
200 stars 7 forks source link

Possible to provide multiview images as inputs? #3

Open ghpkishore opened 6 months ago

ghpkishore commented 6 months ago

Hi, can i give the multiview images by myself and then ask the MVedit to generate the 3Dmodel instead of relying on generating all the different views from a single image and then expecting the model to generate? Is there any specific type of input image angles required for this

ghpkishore commented 6 months ago

This is because the 3D mesh generation capability is amazing! However, I am trying to generate real life objects and can provide input images in different angles.

Lakonik commented 6 months ago

Thanks for the suggestion. MV inputs are definitely compatible with MVEdit, but it's a bit difficult to build a UI for MV images with user defined cameras. We may consider providing instructions or scripts for MV inputs after the code release.

ghpkishore commented 6 months ago

Thank you. Really appreciate it. Even if the entire script is not provided, if you can provide a way to implement it would be really cool. I had tried a few examples of real products and this is the best MESH and texture i have gotten till date. Really hoping for the MV inputs to be made possible. Thanks a lot!

ghpkishore commented 6 months ago

Hi! @Lakonik ! Keep seeing the repo getting updates. So wanted to know if the MV input is implemented.

We may consider providing instructions or scripts for MV inputs after the code release.

Or even if you can help me understand how to provide the multiview camera inputs.

I basically have a "handbag" which I want to create a 3D model, and I have the pictures from all the angles. I tried a single view and the mesh generated based on that with the generated MVs were pretty great! I felt that adding my own MVs instead of the generated MVs would make it far more useful in 3D reconstruction of physical objects.

Lakonik commented 6 months ago

I will probably update the scripts in one or two weeks. Do you have the camera parameters of the MV images? This codebase cannot handle multiview inputs without known cameras yet.

ghpkishore commented 6 months ago

Thanks for the reply on updating the script. I currently don't have exact camera angles. But know approximately, so can use those as inputs, if that suffices. I also can take pics in specified angles if the code can only work with those angles.

ghpkishore commented 5 months ago

Hi @Lakonik I came across https://github.com/xt4d/id-pose which provides the camera angles for the video. Can we use that as input to then generate the 3D view? Will that work? Also wanted to know if the script is updated for the handling on multiview images to generate the 3D model. This i think can solve 3D reconstruction to SOTA level.

Lakonik commented 5 months ago

I would not say MVEdit could potentially be a SOTA 3D reconstruction method. The image-to-3d pipeline is designed to overcome the inconsistency of generated views, instead of faithfully reproduce the generated views. For 3D reconstruction, perhaps you can try this?

ghpkishore commented 5 months ago

Unfortunately the license prohibits any commercial application. The entire issue with gaussian splat is that it is not opensource. Difficult to innovate on a closed source model.