The visualization of image warping

[EDENpraseHAZARD]

Hi Clement, recently I used the your pretrained model dispnet_model_best.pth.tar and exp_pose_model_best.pth.tar to warp the images, but got bad result. I used 5-frame snippets as input, and show the target view frame_2 and its nearby view frame_1 here : target view:
nearby view: Than I got the warped view :

I don't care about distortion which suffered from semi_dense depth prediction since I have the depth ground truth. However, the pose is also inaccurate and the displacement showed between the warped view and the nearby view is too small .I have read the issues of this project and find that the pose groundtruth provided by KITTI is inaccurate . What's more, I notice that the best pose result you showed on the README is test from exp_pose_checkpoint.pth.tar rather than exp_pose_model_best.pth.tar. So I also used exp_pose_checkpoint.pth.tar and dispnet_model_best.pth.tar to do the warping and got worse result: I don't know the reason and think that it may caused by the dismatch of exp_pose_checkpoint.pth.tar and dispnet_model_best.pth.tar for I don't have dispnet_checkpoint.pth.tar.

Moreover, I haved tried many experiments of image warping used SfMLearner but the results were always disappointed. So Clement, have you ever tried image warping and got some good result ? I guess the pose predicted by the posenet is not accurate enough to do image warping.

[ClementPinard]

Hi, I highly suspect the missing normalization. Depth and pose models have not been trained together in the pretrained network folder, and thus warping is not expected to be perfect, but at least it should show some structure, contrary to this LSD soup.

The network expects pictures with pixels in the range [-1, 1].

If you correctly applied the normalization, you might want to see if a network that you trained yourself, even with worse numerical test results than the pretrained one performs better or not, which would indicate some bug in your code.

At a last measure, I'll try to look up what's wrong with my pretrained network.

Clément

[EDENpraseHAZARD]

I check my test code which really do the normalization and the pixels are in the range [-1, 1] Here is the test code: test_warp.zip Moreover I tried to used the model I finetuned with Cityscapes to do image warping, and here is the result : target image: nearby image: warped image: Well you can see, the warped image has a little distortion and shows good structure. However, the wapred image is in alignment with nearby image but not target image which means the pred_pose is inaccurate and the displacement between nearby image and warped image is little. So I guess the pose predicted from posenet is not accurate enough to do image warping

[huhaoyue]

I check my test code which really do the normalization and the pixels are in the range [-1, 1] Here is the test code: test_warp.zip Moreover I tried to used the model I finetuned with Cityscapes to do image warping, and here is the result : target image: nearby image: warped image: Well you can see, the warped image has a little distortion and shows good structure. However, the wapred image is in alignment with nearby image but not target image which means the pred_pose is inaccurate and the displacement between nearby image and warped image is little. So I guess the pose predicted from posenet is not accurate enough to do image warping

Hi, I download your test_warp.zip, but it doesn't have the relevant work. Hope you can provide another copy, thank you very much！(786059151@qq.com)