search

SiTH-Diffusion / SiTH

[CVPR 2024] SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion
https://ait.ethz.ch/sith
MIT License
141 stars 9 forks source link

Unable to generate accurate avatars for new data #21

Open marcusrdlee opened 2 weeks ago

marcusrdlee commented 2 weeks ago

Hi, I set up the pipeline of taking an image, formatting the image, using OpenPose to generate keypoints, etc. to eventually get the avatar meshes. When I use your example image data provided, this pipeline works perfectly and I am able to generate keypoints, back images, and an accurate avatar mesh like your example. However, when I use my own image data, the mesh always turns our corrupted and warped. I am not sure what the issue with this is, can you please help provide advice on how to fix this issue. Here are examples of the results of I am getting.

Not working custom image result:

Image Simon_1

OpenPose rendered results (used these generated keypoints) Simon_1_rendered

smplx result image

Back images Simon_1_all

Final mesh result (not correct) image image

Please let me know what you suggest to fix this issue. I can provide example of how it works correctly on your demo images if needed as well. Thanks for the help.

azuxmioy commented 2 weeks ago

Hi, it looks like the SMPL-X fitting or the resolution issue.

Do you use 1024x1024 resolution when reconstructing the mesh (step 3)? Also, do you have an image showing the overlay of the SMPL-X fitting on the original images?

Thanks!

marcusrdlee commented 2 weeks ago

Hi, Yes I used fit.py to make my images into 1024x1024 resolution before constructing the mesh.

Not sure if I have an overlay of the SMPL-X fitting on the original images. How do I get that? The images I shared previously are all of the data I believe was generated.

Thanks for the help.

azuxmioy commented 2 weeks ago

When you activate the --debug flag in fit.py, the script will generate debugging images.

Also refer to #10. There are several tips to play in the body fitting step. Let me know if you face any issues.