What is I wish to generate the mesh with multiple images ? - I have front view, left view, right view and back view of a chair - how can I create a mesh utilising all these images ?

TencentARC / InstantMesh

InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models

Apache License 2.0

3.06k stars 322 forks source link

What is I wish to generate the mesh with multiple images ? - I have front view, left view, right view and back view of a chair - how can I create a mesh utilising all these images ? #91

Open codebugged opened 3 months ago

codebugged commented 3 months ago

![Uploading hs-seating-liberty-studio-Frechette_Silos_153_RT.jpg…]()

yosun commented 3 months ago

Cam this model be modified to support Multiple ground truth consistent input from the user?

sumanttyagi commented 3 months ago

@yosun yes it can be modified just feed the input image of the same size

sumanttyagi commented 3 months ago

@codebugged the size should be - 320320 into 6 images / 4 images as both 6 and 4 multi view can be used make changes in this line - https://github.com/TencentARC/InstantMesh/blob/main/run.py for line 183 images = rearrange(images, 'c (n h) (m w) -> (n m) c h w', n=3, m=2) # (6, 3, 320, 320) instead of above just load you images using any library which has size of 6 , 3320*320

yosun commented 3 months ago

do you know which coordinate system they are using?

as a test: i tried inputting screenshots from 6 images from a 3D model in the 360%6 azimuth and also 20 and -10 deg elevations, but the results are kinda crap.

sumanttyagi commented 3 months ago

please check , the order to be like images - chair_comfort multi view

yosun commented 3 months ago

Curious - can you share your generated results from that?

yosun commented 3 months ago

screenshot_0 screenshot_1 screenshot_2 screenshot_3 screenshot_4 screenshot_5

sumanttyagi commented 3 months ago

@yosun put it in this size # (6, 3, 320, 320) a big picture with 6 images together

yosun commented 3 months ago

yes but the generated result do not look good! i tried your input and got this!

yosun commented 3 months ago

okay tried it in the order above - slightly better but still not great

yosun commented 3 months ago

sumanttyagi commented 3 months ago

pLEASE CHECK THE AZIMUTH AND degree to be 20 as per paper - deviations from this won't give you good results .


The azimuth, elevation and camera distance of the query image is randomly
sampled from a pre-defined range. The poses of the 6 target images consist of interleaving absolute elevations of 20◦
and −10◦
, combined with azimuths relative to the query image that start at 30◦
and increase by 60◦
for each pose.

so i see one problem your first image to get the better result should be the front view or frontal view it is kind of skewed. So please make first image as front facng then use the above guidelines of azimuths and pose

yosun commented 3 months ago

screenshot_0 screenshot_1 screenshot_2 screenshot_3 screenshot_4 screenshot_5

hmm still kind of a mess