search

Wi-sc / ViewFusion

Code of ViewFusion
Apache License 2.0
46 stars 4 forks source link

Very bad render/mesh - NeuS #1

Closed hadillivvy closed 4 months ago

hadillivvy commented 7 months ago

I tested the 3D reconstruction on the batman sample example provided and the render video and .ply look terrible. Any advice?

https://github.com/Wi-sc/ViewFusion/assets/150967838/c1d30edb-dd8e-44b2-b6fc-0b146f0f5d63

https://github.com/Wi-sc/ViewFusion/assets/150967838/65cde044-6c71-4294-a5e2-87827006193b

Wi-sc commented 7 months ago

Hi @hadillivvy ,

I just tested my code again, and it shouldn't be. Your problem is that the multi-view images are bad, not by the NeuS.

Can you provide more details? What model are you using? Have you checked the dataloader as the initial frame of the video seems broken?

hadillivvy commented 6 months ago

Hi @Wi-sc

I don't think the issue is with the multi-view images because they look fine to me. Attached are a couple of them. To generate these, I used the zero123-xl.ckpt checkpoint and ran the generate_360_view_autoregressive.py script.

input-0 input-7 input-20 input-26 input-32 input-34

Wi-sc commented 6 months ago

Hey @hadillivvy ,

Thanks for the clarification. I have re-produced your issue on the batman image, while I have also run the pokeman images and the reconstruced 3d model looks well. So I think that is not a bug from the code.

I believe the issue here is because the batman is a thin and tall 3d object, with lots of details (cloak, belt, arms, masks, legs). We didn't focus on the NeuS training in our paper, and the training scripts and weights are stolen from SyncDreamer. Maybe you can try to reduce the weight of eikonal loss for NeuS and have a try.

Sorry for the inconvenience. If you have any insights on this issue, just comment here. Let's make the model better!