Question about warp function for time-aware case

[JamesYang110043]

Hi Shiba,

Firstly, I want to commend you on the excellent work you've done. I have a question regarding the time-aware warp function.

In Issue #8, you mentioned that motion-model=dense-flow-voxel corresponds to the time-aware case. However, I noticed the comment # don't know how to calculate for time-aware flow in the def warp_event_from_optical_flow_voxel function.

Does this mean that this function is not available for time-aware cases? https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/warp.py#L396C16-L396C65

I also observed the existence of the def warp_event_from_optical_flow_voxel_optimized function in the warp module. If I aim to implement the results from the paper for time-aware cases, should I use this function?

I would appreciate it if you could provide some clarification on this matter. Thanks in advance. BTW, could you please share the config file for the paper reult?

[shiba24]

Hi @JamesYang110043 thank you for the appreciation.

don't know how to calculate for time-aware flow

tldr you don't need to worry about this comment.

This comment specificly refers to feature calculation. For some other papers such as [Shiba et al. Sensors 2022], [Shiba et al. Advanced Intelligent Systems 2023], I implement the warp function such that it returns both warped events and some feature to tell if the warp causes collapsing or not. (see https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/warp.py#L163 and https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/warp.py#L179-L180) In this ECCV paper, these features are not necessary. And the warp is implemented as expected, which you should be able to use it as it is.

[shiba24]

warp_event_from_optical_flow_voxel vs. warp_event_from_optical_flow_voxel_optimized

Please see the documentation for this function warp_event_from_optical_flow_voxel_optimized: https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/warp.py#L402-L405

You could try from motion-model=dense-flow-voxel, as I think this is more stable in terms of implementation if I remember correctly. It consumes some memory proportional to the time_bins. At some point when you have memory issues, you could think of using warp_event_from_optical_flow_voxel_optimized. I suppose the optimized version function should work without problems, but if you face any, please post here. Thank you.

[shiba24]

share the config file for the paper result

Which dataset and sequence do you mean?

[JamesYang110043]

I suppose the optimized version function should work without problems, but if you face any, please post here. Thank you.

I think sharing the MVSEC dataset and its 'indoor_flying1' sequence should suffice, as the configurations for the other sequences are likely the same, correct?

[shiba24]

MVSEC indoor_flying1 is there: https://github.com/tub-rip/event_based_optical_flow/blob/main/configs/mvsec_indoor_no_timeaware.yaml

[JamesYang110043]

MVSEC indoor_flying1 is there: https://github.com/tub-rip/event_based_optical_flow/blob/main/configs/mvsec_indoor_no_timeaware.yaml

Thank you very much for your patient explanations.

I have another question: In mvsec_indoor_burgers.yaml the time-aware parameter is set to Truth. But why isn't the motion_model parameter set to the time-aware case dense-flow-voxel as you mentioned earlier?

[shiba24]

I admit that it's not the cleanest implementation but please check https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/solver/base.py#L215

[JamesYang110043]

Thank you again.

I admit that it's not the cleanest implementation but please check

https://github.com/tub-rip/event_based_optical_flow/blob/756dfd0125ca0b9092d207e7aed982b763be1d32/src/solver/base.py#L215

Thank you again!