Compatibility with PWC-Net?

[SwagJ]

Hi @gengshan-y ,

Thank you for your great work. I just have a question. Have you tested the compatibility of your implementation with PWC-Net?

[gengshan-y]

Hi, I haven't tested, but I think the inference pipeline should be able to work with any flow input.

Note the expansion layers only depend on optical flow and image features (from VCN image encoder). One could replace line 499-516 with PWC-Net and make sure the flow fields matches the resolution of original VCN outputs (where flow is computed on 1/4 image resolution).

[SwagJ]

Hi @gengshan-y ,

Thank you very much for your reply. Indeed, I have tried PWC-Net with your expansion pipline. However, I just have questions about your data augmentation.

1. I noticed that in your augmentation. you used PCA Aug, Chromatic Aug and random rotation, scaling and cropping. But, the input image pair you used are just original image with partial cover, right? Then your GT exp and mid are augmented? Will this be inconsistent?
2. How would image partially covering benefit the performance of your expansion pipeline? For me, your expansion performance relies on good flow prediction and quality of correlation at top level. So I am just curious about this.

Best,

[gengshan-y]

From what I remember,

1. The ground-truth optical expansion is computed on the fly from ground-truth optical flow (not the estimated ones). So there wouldn't be inconsistency wrt expansion.
2. During training, the input images and flow are augmented together (color+geometric+occlusion augmentation). Here, only the geometric augmentation is applied to optical flow. One could think of it as changing camera intrinsics for both frames, which does not affect the perceived 3D geometry. Therefore I don't see inconsistency wrt motion-in-depth.
3. At training time, the occlusion augmentation avoids getting over-fitted to clean flow measurements. (Note the ground-truth flow is not changed by such occlusion augmentation)

[SwagJ]

Hi @gengshan-y ,

Thank you for your reply. Indeed, 3D geometry is irrelevant to image augmentation. Just have one last question. Is that possible for your to provide the VCN model's AEPE on Synth Driving dataset with pretrained weight flow-things, if you happens to observe that?

Best,

[gengshan-y]

Hi, I don't have that number. It would be straightforward to modify eval_flow.py to support evaluation on synthetic driving dataset, but I don't have cycles to work on it right now, thanks.

[SwagJ]

Hi @gengshan-y ,

Thank you for your reply. I will try that!

Best,