Numeric instability in warping module

[tomas2211]

Hello,

I am not sure if this bug affects anything significantly and whether it is connected to precise GPU model.

I am getting strange patterns when testing the warping layer with down-sampled flow. I use Sintel - alley_1, frame 5 for testing with the following code:

warping_layer = WarpingLayer()

div_flow = 0.05
_, _, h, w = fl_gt.shape

im1 = tf.interpolate(im1, (200,500), mode='bilinear', align_corners=True)
im2 = tf.interpolate(im2, (200,500), mode='bilinear', align_corners=True)
fl = tf.interpolate(fl_gt, (200,500), mode='bilinear', align_corners=True)

im2_warp = warping_layer.forward(im2, fl * div_flow, h, w, div_flow)

And this is the outputs I get:

Im1

Im2_warp

The warped image contains weird patterns that look to me like numeric instability issue. When no down-scaling takes place, the patterns do not emerge.

I tracked down the problem to masking in the warping module. It seems that the patterns are caused by the condition on line 108. I suppose some correctly-warped pixels contain some number close to 1.0 but lower than one. https://github.com/visinf/irr/blob/3c0468b1b9d24d43d3f758be709b912f58b94229/models/pwc_modules.py#L106-L108

This can be easily fixed by replacing the line with:

mask = (mask >= 0.9999).float()

which makes the patterns disappear.

When testing the optical flow estimation with 'SintelTrainingFinalValid', I do not get a measurable AEPE decrease with the pre-trained model after the fix.

[hurjunhwa]

Hi,

Thanks for sharing your finding! Yes I think it's better to follow your suggestion so that it doesn't ignore the correctly warped pixels (or feature maps) and fill-in the black instead:

mask = (mask >= 0.9999).float()

Also the original PWC-Net's source code follows the same: https://github.com/NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py

However, I will leave my implementation as it is.. as it's already published and should stay as it is. For the future research, it seems better to use the revised one :)

Thanks for sharing!

Best, Jun

[hurjunhwa]

Interestingly, using

mask = (mask >= 0.9999).float()

degrades the accuracy a little bit ( less than 1% ).

I guess that this weird random pattern due to the numerical instability works as a noise (or can be regarded as occlusion) that improves the accuracy a little bit.. :)