inconsistent and incorrect scale in predicted depth

[ChauChorHim]

Hi there, thanks for sharing your codes. I am trying to train and predict my own dataset. The predicted depth jpeg image looks fine in eye. However, when I try to reconstruct the depth image into pointcloud in camera coordinate, the points are inconsistent in the same object and in the whole scene.

For example, the upper pole on the right are stretched, which is not obvious in the jpeg image. Also, the scale of the car is not normal.

I think one of the reason is that the scale of depth is not correct locally and globally.

Have your group noticed these issues? I am really appreciated if you could give me some advices!

RGB image

Depth image

Pointcloud - the poles are twisted

Pointcloud - the car is streched

[daniyar-niantic]

Hi. Do you use single images for prediction, or multiple images?

[iariav]

having a similar issue. single image prediction.

[daniyar-niantic]

With single image prediction this is probably acceptable quality of depth maps. You have different intrinsics then at training time, and there are no additional images to resolve depth ambiguities.

[iariav]

thank you for your reply. But why do you assume different intrinsics? if I train and test on images generated from the same camera rig, it's the same intrinsics, right?

[daniyar-niantic]

Oh, sorry I thought this is trained on KITTI. You are right, the intrinsics would be the same. I would suggest to make a multi-frame prediction and see if local/global scale is correct. If it too bad, then there may be a problem with data loading/processing or the model has not converged.