search

tub-rip / event_based_optical_flow

The official implementation of "Secrets of Event-based Optical Flow" (ECCV2022 Oral and IEEE T-PAMI 2024)
GNU General Public License v3.0
137 stars 14 forks source link

Questions about warp func. #8

Closed Hazeliii closed 1 year ago

Hazeliii commented 1 year ago

Thanks for your amazing work. I have two questions:

  1. There are different “motion model” when warping events, such as “dense-flow”, “dense-flow-voxel”. What do they represent respectively? How should I choose?

  2. When applying to unsupervised-learning methods, as you mentioned in the paper, the EV-FlowNet, I’d like to take the event volume as the input, as proposed in Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion by Alex Zihao Zhu. So I need to center crop the event volume to 256x256, then the predicted flow is 256x256x2. But the resolution of event sensor is 346x256.What should I do in the warp function? Do I need to choose the events triggered at x∈[45,300] and y∈[2,257]? Because the predicted flow only covers those regions. Am I right?

shiba24 commented 1 year ago

  1. Check https://github.com/tub-rip/event_based_optical_flow/blob/main/src/warp.py#L263 and https://github.com/tub-rip/event_based_optical_flow/blob/main/src/warp.py#L315 . dense-flow-voxel is the time-aware case.

  2. That's a more general question about DNN-based methods. In the training people usually do random cropping, and crop only the center in the inference.

Hazeliii commented 1 year ago

I am sorry that I don't understand what's the meaning of " crop only the center in the inference". When I do crop in the DNN-based method, should I deal with the events(choose those events in the cropped region) before warp or do the crop directly to the IWE?

shiba24 commented 1 year ago

@Hazeliii I think that cropping after warp is not possible because the output resolution is 256x256. One cannot warp events where there is no flow prediction.