search

CAOR-MINES-ParisTech / colibri-vr-unity-package

This is the Unity package for COLIBRI VR, the Core Open Lab on Image-Based Rendering Innovation for Virtual Reality.
https://caor-mines-paristech.github.io/colibri-vr
Other
50 stars 11 forks source link

Ghosting artifacts in ULR for virtual scenes #12

Open rumblecoder opened 3 years ago

rumblecoder commented 3 years ago

Hi, I have a question regarding the implementation of the unstructured lumigraph rendering: Why can I see ghosting artifacts for virtual scenes created with the provided acquisition interface using the high quality (non-optimized) ULR shader? I would expect that the UL rendering is near perfect because everything I provided (images, depth, cameras and mesh) is ground truth. There should not be any inaccuracies that could explain these artifacts. I also checked the original ULR paper but could not find any hints for this problem (except for the proxies maybe?). Do you have any idea why this is happening?

DinechinGreg commented 3 years ago

Hi, Thank you very much for your message! And sincerely sorry for not answering sooner.

While ULR is quite great, some issues do remain that could explain the presence of ghosting artifacts. In particular, if your problem is that colours from a foreground object are projected onto a background object, then I believe the issue may be with the visibility check. While I don't recall it being included in the algorithm described in the original paper on ULR, the authors of the paper do mention that "With a highly accurate geometric model, the visibility of any surface point relative to a particular source camera can also be determined. If a camera’s view of the point is occluded by some other point on the geometric model then that camera should not be used in the reconstruction of the desired ray". So if we have a foreground object occluding the background in the captured photographs, then a visibility check should be performed to prevent artifacts from appearing on the background. We set up a basic version of such a check in the toolkit, which is why we store per-view depth data as well as the global mesh. We implemented this in the CheckPointNotOccluded method in ULRCG.cginc.

To compute our visibility check, we notably rely on a modifiable factor, named _DepthCorrectionFactor, which can be modified interactively in the GUI of the Rendering component (Depth. corr. under Blending method). The current implementation is far from perfect, but giving this factor a smaller value could perhaps help with your issue. Could you please try modifying the value of this factor in the GUI during rendering, to see if it helps reduce the amount of ghosting artifacts ?

Thanks again for your feedback, feel free to keep me posted on the issue!

rumblecoder commented 3 years ago

Thank you very much for the explanation.

Decreasing the depth correction factor removes indeed the ghosting artifacts. Unfortunately new artifacts like small holes and strange lines appear now in the rendering: colibrivr_depth_correction_compare

I think I need to check the blending steps in detail and modify them to my needs.