search

SUDO-AI-3D / zero123plus

Code repository for Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model.
Apache License 2.0
1.56k stars 108 forks source link

Is there an approach to remove the gray background purely? #64

Closed bluestyle97 closed 4 months ago

bluestyle97 commented 5 months ago

Thanks for your valuable work! But I'm wondering why zero123++ generates images with a gray background, which leads to much inconvience when feeding the synthesized multiview images to image-to-3D reconstruction pipelines such as OpenLRM or NeuS (Also, Instant3D if it's open-sourced in the future), since most of these methods require a white background for the input images. Using the rembg package to remove the gray background leads to inconsistent and wrong segmentations especially when there are image regions close to the gray color. So I'm wondering if there is an approch to remove the gray background purely and generate consistent white-background images? Also, I'm curious about whether you directly feed the gray-background images to the 3D diffusion model as conditions as described in the One2345++ paper?

Time-Lord12th commented 5 months ago

Same question here. Will the image be preprocessed as a gray background during training? image

eliphatfs commented 5 months ago

Same question here. Will the image be preprocessed as a gray background during training? image

Yes.

eliphatfs commented 5 months ago

The background can be removed with SAM, better than rembg; you can check the demo code for details. The IoU is in general about 96%, which is good but admittedly not perfect. We are planning to release a new version of the model that deals with the issue, but before that we can do few about it.

And yes, for One2345++ we feed gray background images to the 3D diffusion model during training, so there is no gap here.

eliphatfs commented 4 months ago

We released v1.2 that has a normal-based approach for alpha mask prediction. Note that the approach is not suitable for semi-transparent objects, only pure opaque ones.