Per-Pixel Minimum Reprojection Loss in backward sequence

[alwynmathew]

The monodepth2 considers the middle frame t as the target frame and makes a forward sequence warps t-1->t, t+1->t. Since the target is the same for the photometric error, the minimum is considered for both the warps. If we also perform a backward sequence t->t-1, t->t+1 where the target can be t-1 and t+1. Taking min for the photometric error would be wrong. Also averaging them would yield a higher loss. Kindly, let me know your thoughts on this.

[daniyar-niantic]

Yes, if I understand you correctly, then I agree with you.

One important reason we map from t-1->t and t+1->t is due to low computation effort and dense reconstruction of backward warping. Check out these slides to learn more about forward and backward warping: https://www.cs.tau.ac.il/~dcor/Graphics/pdf.slides/rotation2017.pdf

To do warping from t (such as t->t-1 and t->t+1) using depth map of frame t you need to do forward warping. So, the images that you are reconstructing would be t-1 and t+1, which won't align. You can still compute photometric error between reconstruction at t-1 and the actual frame at t-1 (similarly for t+1), but that would be outside the scope of this project.