How to visualize flow that model inferenced ?

[zhishao]

flow, mask, merged = self.flownet(imgs, scale_list) flow must be the flow between two images. it's shape (bs, 4, H, W) , how to visualize it ? like this: and how to generate the flow groudtruth ? Thx !

[HamidGadirov]

You can return flow vectors from RIFE.py and then flow[2] is what you need for visualizing the flow of the last IFNet block.

[hzwer]

Hi, this is our visualization of mask. You can visualize flow[:, :2, :, :] using https://github.com/tomrunia/OpticalFlow_Visualization. And there is no real ground truth for videos, you may use https://github.com/princeton-vl/RAFT to generate pesudo labels.

[zhishao]

@hzwer Thx for anser. And in IFNet_HDv3.py, the model output: flow_list, mask_list, merged, what do they represent?

[hzwer]

@zhishao flow_list represents flow predictions by each IFBlock, 4 channel, Ft->0, Ft->1 mask_list represents fusion mask predictions by each IFBlock, 1 channel, by sigmoid merged represents interpolated image results by each IFBlock, 3 channel, rgb

[zhishao]

@hzwer Thx a lot! Many difference between IFNet_HDv3.py and IFNet.py，i need time to get it clear.

[zhishao]

@hzwer Hi, what is the loss_cons in RIFE_HDv3.py ?

[hzwer]

@zhishao It's the distillation loss of RIFE, another name of loss_distill.

[zhishao]

@hzwer Hi, i tried on some flashing cases, the dark scenes with flashing lights，the flow not work well, if i want to make it better , should i add the data and retrain the model ?

[hzwer]

Will properly increasing the brightness of the picture improve the result? Increasing data should be an effective method.

[zhishao]

The RIFE model distilled form Lite-flownet ? If I add my custom dataset and train , what should I do ? Follow the Lite-flownet train steps ? Or just train RIFE with custom data ?

[hzwer]

@zhishao Hi, we don't need liteflownet after some very early versions. If your want to use this trick, you may refer to IFRNet's ReadMe. https://github.com/ltkong218/IFRNet

[zhishao]

So the flow is not a supervision signal ? Not a groundturth ? Just a mid product of the model ? Real work is to fuse three resolutions flow or warp_image ? The real supervision signal is still a frame(t=0.5) ?

[hzwer]

@zhishao For real video, the optical flow estimated by the existing model can somehow help the frame interpolation model, but there are many limitations. So we gradually iterated the scheme to apply only intermediate frames for supervision.