search

med-air / EndoNeRF

Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery
https://med-air.github.io/EndoNeRF/
173 stars 14 forks source link

not_zero_canonical #18

Open smoreira00 opened 1 year ago

smoreira00 commented 1 year ago

What does exactly this 'not_zero_canonical' parameter do? In which situations do I have to set it to True/False?

yuehaowang commented 1 year ago

When not_zero_canonical is set to False, the canonical frame is set to the first frame (t=0), i.e., all the displacements of the first frame are set to zeros. When set to True, the canonical frame is not explicitly specified.

Please refer to the related code: https://github.com/med-air/EndoNeRF/blob/2d4546f58970b7cb3bb2465daee6c36c4f68f3cb/run_endonerf_helpers.py#L121-L122

In general, this hyperparameter only affects the rendering/reconstruction of the first frame.

smoreira00 commented 1 year ago

So when we set not_zero_canonical to True, does the EndoNeRF randomly chooses a frame to set as the canonical frame? Or it doesn't set any frame as the canonical one?

yuehaowang commented 1 year ago

When not_zero_canonical is set to True, the model does not take any frame as the canonical space but just predicts displacement fields (generally non-zero) for each frame.

smoreira00 commented 1 year ago

And when not_zero_canonical=False, why does the rendering/reconstruction of the first frame get so affected?

yuehaowang commented 1 year ago

This is because the canonical field is supposed to contain all the appeared areas in the whole video. Since there is no strong regularization of the predicted displacements, those new areas in the live frame might be remapped to random locations in the canonical field, resulting in some artifacts in the first frame.