Finetune Problem Caused by KITTI Sparse GT

lmb-freiburg / flownet2

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

https://lmb.informatik.uni-freiburg.de/Publications/2017/IMKDB17/

Other

1k stars 318 forks source link

Finetune Problem Caused by KITTI Sparse GT #153

Closed Elena-ssq closed 6 years ago

Elena-ssq commented 6 years ago

Hi,

I'm trying finetune the network on KITTI dataset while the results are greatly affected by the sparse ground-truth. I did notice that points with unavailable gt should not be considered during loss computing, but I didn't find a parameter to mask them out.

Is there any suggestion on how to achieve the finetune on KITTI?

Tkx

nikolausmayer commented 6 years ago

Hi, The loss layer ignores NaN values (see https://github.com/lmb-freiburg/flownet2/blob/master/src/caffe/layers/l1loss_layer.cu#L22), but the invalid pixels in KITTI are "zero", not "NaN" (it's a PNG format which cannot represent that, I think). You should be fine if you take that into account when converting your data.

Elena-ssq commented 6 years ago

Hi,

The problem is solved by following your suggestion.

Thanks again!

Elena-ssq commented 6 years ago

Hi,

The results are really blurred after finetune by killing (set to zero) all diff without GT. The parameter normalize_by_num_entries in l1loss layer was assigned to true, and remained others as false.

Any clue why this happens? Thanks a lot!

nikolausmayer commented 6 years ago

Did you use the S_fine solver schedule?
Did you adjust the crop size for KITTI?
IIRC normalize_by_sum_entries is only used to calculate a normalized EPE value.
Considering that KITTI has no groundtruth at object boundaries, a little bit of blurriness is expected. If it's really bad, something else is wrong..

Elena-ssq commented 6 years ago

Yes, I adopted the S_fine solver schedule, while I did attempt to reduce the base learning rate to 3.125e-6.
I resized all KITTI images into 1216x320 before sending to the net because the net requires for consistent image size. However, I didn't figure out the relationship between the original image size and the cropped size, so I set the cropped height and width equal to the image size, which is 320x1216. Any suggestion on the cropped size?
I see, thanks for the note.
The blur is quite obvious almost among the entire image, so I don't think it's normal.

Please help.