search

SUDO-AI-3D / zero123plus

Code repository for Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model.
Apache License 2.0
1.56k stars 108 forks source link

Depth conditioning is not working #52

Open AtlantixJJ opened 7 months ago

AtlantixJJ commented 7 months ago

I am trying with the following example with a given depth prior.

Screenshot 2023-11-25 at 9 14 58 PM Screenshot 2023-11-25 at 9 15 39 PM

The results are quite out of expectation.

Screenshot 2023-11-25 at 9 16 07 PM

On the other hand, if no depth conditioning is used, the results are somewhat closer:

Screenshot 2023-11-25 at 9 16 34 PM

Do you have a clue what's wrong with the depth conditioning part?

pragyavaishanav commented 7 months ago

I have similar issues. 4_image (8) depth_combined (1)

Screen Shot 2023-11-27 at 10 20 50 AM

To me it seems, it is assuming an underlying geometry of the model when we pass the front side image. Then it try to recreate geometry and then draw it on that geometry instead of taking the our custom depth maps into account properly. Look at my results. It was drawing two arms as if reading my depth and its OWN (concept of depth) depth and then try to work with both somehow but terribly failing.

pragyavaishanav commented 7 months ago

At least I know why you (@AtlantixJJ ) have issue. I was having similar problems as yours. I got rid of the scheduler that they are using. Once scheduler wasnt used, then it was drawing the input image on the depth maps (even though still weirdly). I had added an issue related to the same previously here, https://github.com/SUDO-AI-3D/zero123plus/issues/49

eliphatfs commented 7 months ago

image The example script should have an output similar to this figure. If you cannot reproduce the example script, please send me your diffusers, torch version and GPU setup, I guess there are compatibility issues instead of issues with the model.

eliphatfs commented 7 months ago

If you are good with the example but failing with your input, maybe you could tune the controlnet_scale parameter. In 2D SD controlnets this also needs to be tuned. Another thing is that maybe the input is not matching the requirements of the depth condition. Please see if your scales or background is different to the example depth input.

odusseys commented 4 months ago

This is because of the scheduler used in the example code. Just remove it and the results will be correct.