search

Daniel-xsy / RoboBEV

RoboBEV: Towards Robust Bird's Eye View Perception under Common Corruption and Domain Shift
https://daniel-xsy.github.io/robobev/
335 stars 32 forks source link

Robustness of 4DModel? #17

Open dingmiaomiao opened 1 month ago

dingmiaomiao commented 1 month ago

Thank you for your work. I noticed that RoboBEV tested methods like BEVFormer using past frames. The RoboBEV dataset has only sample corruption. If the past frames are fine but the current frame is corrupted, it won’t reflect the true robustness of the 4D models. In real situations, both past and current frames should have the same corruption. How do you test the 4D model? Can you provide more details? Thanks!

Daniel-xsy commented 1 month ago

Thanks for your interest.

BEVFormer stores a history BEV feature inside the model, we follow the same evaluation protocol as testing on the original scenes dataset. Therefore, all the input (including past frames and current frames) are both corrupted data. This should reflect the real corruption situations.

It should be noted that since RoboBEV only corrupts sample data (not sweep), we use corrupted sample data as previous frames instead of using corrupted current sample and previous clean sweep data.