search

graphdeco-inria / gaussian-splatting

Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Other
14.79k stars 1.93k forks source link

Trained with Segmented Images, Results with Many Additional Black Ellipses Around the Object. #1039

Open VichyTong opened 3 weeks ago

VichyTong commented 3 weeks ago

Hi everyone,

I'm trying to train a .ply model for a single object, e.g., a sofa.

Here are the steps I took:

  1. Segmented the images of the sofa with SAM2. (Since some people mentioned there is an edge translucency issue, I also tried BiRefNet but got the same result.) Sample segmented image:

  2. Provided the segmented images with black backgrounds as .png files to Colmap.

  3. Trained the model using the Colmap dataset output.

The issue is that there are some additional black ellipses around the sofa.

There might be two ways to solve this issue:

For the first method, I haven't yet figured out the reason for the ellipses. For the second method, I've tried filtering by distance to the object center, by the tail end of the xyz axis, and by color, but none of these experiments worked. The result is either that the black ellipses remain, or some necessary points are removed.

I also find #542 seems related to my issue, but what I need is a model without the information in the mask.

If anyone has experience with this or suggestions on how to effectively remove these unwanted black ellipses without affecting the main object, I'd really appreciate any guidance. I'm open to trying different approaches or specific configurations in the segmentation or post-processing steps. Thanks in advance for your help!

AsherJingkongChen commented 3 weeks ago

I saw this image has alpha channel and transparent background, which may lead to undesired background colors.

You can try converting the PNG to JPEG manually before training. The background should be black.

VichyTong commented 3 weeks ago

@AsherJingkongChen, thanks for the suggestion!

I converted my dataset from PNG to JPG (dataset link) and trained the model. It seems the background color still has some messy points.

image

However, the black points are different from the results from PNG dataset. I think it's matter to set images to JPG. I'm going to try to find out the reason of generating these annoying balck dots today.

Update: If I train the model with the original background, the result seems to be perfect:

{C3486156-F603-462C-B34B-AACE6955F348}
AsherJingkongChen commented 3 weeks ago

@VichyTong

The original Gaussian Splatting does not respect the opacity on PNG images. It does not learn anything from the transparent or fake background. Moreover, it may move splats onto the clean background.

It converts images from any format to RGB in two phases: One is COLMAP preprocessing, and another is Pillow's image reader. Both processes are destructive for RGBA or HDR images.

It is recommended to find other implementations to actually learn from your images. I found this issue is pretty common (Transparent image for training splats), so I'll figure out some solutions in the future, too.

VichyTong commented 3 weeks ago

@AsherJingkongChen

It converts images from any format to RGB in two phases: One is COLMAP preprocessing, and another is Pillow's image reader. Both processes are destructive for RGBA or HDR images.

This explanation makes sense for my results. Thank you for you help!