search

XiangZ-0 / GEM

Pytorch implementation of ICCV'23 paper "Generalizing Event-based Motion Deblurring in Real-World Scenarios"
Apache License 2.0
30 stars 2 forks source link

Question regrading the metric of LEVS and Motion-ETR. #4

Closed chenkang455 closed 8 months ago

chenkang455 commented 8 months ago

Hi, senior @XiangZ-0 . Thank you for your great work. I have a question about performance metric calculations.

The paper mentions that the metrics are calculated based on sequence reconstruction, but for blur decomposition tasks like LEVS and Motion-ETR, since the motion direction cannot be discerned from the blurred image, there exist sequences with completely opposite directions but identical blur. (moving right and moving left objects for example). I'm curious about how these two metrics are calculated. (Perhaps I have a misunderstanding about the blur decomposition task.)

Another question is about the deblurring results of LEVS. It seems like there is color distortion. Is this normal? image

I would be very grateful if you could clear up my confusion.

XiangZ-0 commented 8 months ago

Hi chenkang455, thanks for your interest in our work and your good questions. Your understanding is correct. For frame-based deblurring methods, they often struggle to handle large blur due to the missing motion information (which we often refer as "motion ambiguity"), so they tend to produce results with wrong motion trajectories (like the opposite case you mentioned) even with the help of trajectory modeling (Motion-ETR). This is also why frame-based methods usually suffer from performance degradation under large blur. With the aid of events, the motion ambiguity can be alleviated, leading to more precise motion results for event-based methods. The computation of metrics in our paper is based on the timestamps of results. For both frame-based and event-based methods, we will generate latent sharp images at the same timestamps and then compute the metrics.

For the color distortion issue, I assume that is related to specific model architecture and training strategy. For frame-based approaches, Motion-ETR is mainly based on warping and thus does not suffer from color problems, while LEVS is mainly based on image synthesis, which might be the main reason. Another possible cause is that LEVS is trained with much smaller blur, and thus might produce unwanted results under large blur. Hope this helps :)

chenkang455 commented 8 months ago

Thanks for your rapid reply. Your answer has greatly helped me understand this job. Thanks a lot!🥳