Unable to finetune

[ckkelvinchan]

Thank you for your insightful work!

When I train a network (different architecture to your paper) using forward warping, I found that the model crashed when fine-tuning together with PWC-Net (reflected by both the training loss and validation PSNR). The interpolation network is trained before fine-tuning with PWC-Net.

To check the problem, I tried a very simple model: Given two images x1, x2,

1. Compute the flow from x1 to x2, and multiply by 0.5.
2. Forward warp x1 using average splatting (to disregard the effect from the metric).

In this model, only the PWC-Net is trainable. However, the performance is strictly worse than using the pretrained PWC-Net. I have tried different learning rate and none of them works.

Do you have any ideas about the problem? Thanks again!

[ckkelvinchan]

I have tried an even simpler model using PWC-Net:

1. Compute the flow from x1 to x2
2. Forward warp x1 (using average splatting)
3. Compute the loss between x2 and (warped x1)

When I use backward warping, it works fine (the loss is at a magnitude of 1e3 to 1e4). But when I use forward warping, the loss goes up to 1e5. I have tried 1e-5 and 1e-6 for the learning rate, but both of them do not work.

Do you have any ideas about this? Thanks!

[ckkelvinchan]

Furthermore, I found that there is an issue about the warping. I am not sure whether it is related to the aforementioned problem:

When I use the below code, the expected output should be all 1, since the flow is 0 and hence the output should be identical to the input. However, the output is 0.0909. And when I replace the 1e-8 in line 247 with different values, different results are obtained:

1e-8 => 0.0909
1e-7 => 0.5000
1e-5 => 0.9901
1e-3 => 0.9999
1e-1 => 1.0000

[sniklaus]

I am afraid that I am unable to debug your issue since you did not share your source code. My best guess is that you are enforcing a loss in invalid regions (regions where nothing maps to).

The issue that you raised is as expected to avoid divisions by zero as per the following line.

https://github.com/sniklaus/softmax-splatting/blob/ff9f34ad3aee391a5486be284a3038bd741ec1a8/softsplat.py#L352

Alternatively, you could replace this line with a mask and only divide by non-zero denominators.

[ckkelvinchan]

Hi Niklaus, thanks for your reply!

Regarding the comparison about backward warping and forward warping. The model I used is attached as follow:

I adopted your pretrained PWC-Net, and the loss is simply the L2 loss of the output and zero (i.e. || (img_h - img_n) - 0 ||_2). In this case, I think the loss should decrease (or at least does not increase). However, I found that the loss keeps increasing. Please let me know if any more information is required. Thanks a lot for your help.

Regarding your reply about enforcing a loss in invalid regions, may I ask whether I should multiply the output of forward warping to a mask so that the loss will not propagate back to the regions where nothing maps to?

Thanks again!

[sniklaus]

I am afraid that you still didn't provide your code. You provided partial screenshots of it, but that doesn't allow me to run and debug this. I also do not know what goal you are trying to achieve. Maybe backward warping is the preferred approach for your case. I also do not know what you have tried already. It is, for example, completely expected that the loss keeps increasing if your learning rate is too high. Ultimately, I am afraid that my time is limited and I am sorry to say that I cannot do your research for you.

The safest would be to only enforce the loss at valid pixels (pixels that receive a contribution), so instead of multiplying the output you do something more like estimate[mask] - truth[mask].

[ckkelvinchan]

Thanks a lot for your reply. Maybe I will try the mask first. Thanks again.

[sniklaus]

Good luck and thank you for being interested in our research! :+1:

[mrchizx]

@ckkelvinchan Hi, I have encountered the same problem of not able to finetune the PWC-Net, have you solved it?