search

Dayan-Guan / DA-VSN

Code for <Domain Adaptive Video Segmentation via Temporal Consistency Regularization> in ICCV 2021
MIT License
43 stars 8 forks source link

Optical flow is not used for propagating #2

Closed EDENpraseHAZARD closed 2 years ago

EDENpraseHAZARD commented 3 years ago

Hi, author. I have two questions. The first is I find that you didn't use flow to propogate previous frame to current frame. You just use it as a limitation that the pixel appeared in both cf and kf will be retained. This is unreasonable. image And I refine the code using resample2D to warp kf to cf, but the result only improve a little.

The second question is that I try to train DAVSN for 3 times on 1080Ti and 2080Ti following the setting you gave, but I only get 46 mIoU which is 2 point less than you.

EDENpraseHAZARD commented 3 years ago

The experiment is based on GTA V.

Dayan-Guan commented 3 years ago

Hi @EDENpraseHAZARD , thanks for your information. For the first question, this issue has been corrected as follows: image. For the second question, I tested the trained models and the output is below: ''' Current best model: /home_nfs/dayan/uda/adv/experiments/snapshots/davsn_viper2city/model_36000.pth Current best mIoU: 47.85 [86.8, 36.7, 83.5, 22.9, 30.2, 27.7, 83.6, 26.7, 80.3, 60.0, 79.1, 20.3, 47.2, 21.2, 11.4] ''' The model is trained on one Nvidia Tesla v100. If you could not get ~47.8 mIoU, please try to change the cfg.TRAIN.SAVE_PRED_EVERY from 1000 to 200 (or 100) for dense evaluation.

EDENpraseHAZARD commented 3 years ago

The flow is cf to kf, so you should also correct x_flow : x_flow = int(torch.round(x - flow_cf[:, 0, y, x][0]))

EDENpraseHAZARD commented 3 years ago

In common VSS, we often get inverse flow as show in the code. Optical flow is always fractional, so the flow is from current to previous, and the common way to propogate is Resample2D(bilinear interpolation). image

Dayan-Guan commented 2 years ago

Hi @EDENpraseHAZARD , I have fixed these issues. Please check thanks! I have used this revised code to train the models and updated the trained models. The performance is better (47.9 mIoU in viper2city task and 50.6 mIoU in syn2city task) as these issues are corrected.