Problem in rendering - Githubissues

dcharatan / pixelsplat

[CVPR 2024 Oral, Best Paper Runner-Up] Code for "pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction" by David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann

http://davidcharatan.com/pixelsplat/

MIT License

830 stars 56 forks source link

Problem in rendering #46

Closed Pixie8888 closed 6 months ago

Pixie8888 commented 6 months ago

Dear author,

I am using pixelsplat on my own datset. But the result is very bad, like this below. I don't know what caused the pattern of lower left and right. Also, why there are so many black dots? I set background color to black, and I checked the image value is within 0~255. Do you have any suggestions for debugging? Thank you very much!

dcharatan commented 6 months ago

The input image range should be 0 to 1 (float). Because of the artifacts in the bottom of the scene, I think this could be a problem with the input data format. What's the dataset you're using? And are you trying to train on the dataset or evaluate a pre-trained checkpoint on new data?

Pixie8888 commented 6 months ago

I am using scannet. The input image is normalized to 0~1 (float). I uses the provided intrinsic and extrinsic by scannet. The intrinsic is normalized by image width and height. I am training a new model on it.

dcharatan commented 6 months ago

Some more things to try:

Double-check that the intrinsics and extrinsics are correct by visualizing the epipolar lines as described here.
Confirm that the context views have enough visual overlap to make learning feasible.
Overfit on a few scenes that seem like they should work (see #40).
Try out different near/far disparity thresholds and visualize the resulting spatial distribution of the buckets (it's easier for the network to learn when more of the buckets are overlapping)

I might be able to come up with some more ideas if there are more details you can provide. I haven't seen the artifacts/black dots while training on RE10k and ACID though, so I'm not sure what causes those. Good luck!

Pixie8888 commented 6 months ago

Dear author, I found the main reason is that I mistakenly set the opacity to be always 1 and downsampled the src image to be smaller than target image that causes these artifacts. Thank you very much for your kind help!