Closed JasonQSY closed 6 months ago
Hi! Apologies for the late reply.
For 1. It does look like the GT and preds are swapped -- @Ainaz99 can you confirm? I think the effect here is simply that the loss is applied even on pixels that should be masked. Otherwise the VN loss should be correct since it is symmetric except for masking based on GT depth. My guess is that this didn't affect training much since we only selected the quartile of points with the smallest VNL on which to apply the loss (same as the midas paper).
For 2. I believe the fx and fy correspond to NDC and not screen space and so this should be correct. The naming is perhaps a bit confusing here and please correct me (or @Ainaz99 correct me) if I'm wrong.
Hi,
It looks like the training of depth estimation has some issue.
However in the forward function https://github.com/EPFL-VILAB/omnidata/blob/main/omnidata_tools/torch/losses/virtual_normal_loss.py#L151
The focal length seems to assume the camera width is 1 or it's fov. However, it's actually in pixel space. If you look at the code in virtual normal loss which projects depth back to point cloud, it's in screen space. https://github.com/EPFL-VILAB/omnidata/blob/main/omnidata_tools/torch/losses/virtual_normal_loss.py#L44