search

uzh-rpg / DSEC

MIT License
135 stars 17 forks source link

Optical Flow's alignment with Images #69

Open AbdessamadElK opened 4 months ago

AbdessamadElK commented 4 months ago

Dear authors,

I am interested in a new architecture where it is important that Optical Flow be aligned with Images. However, there is always a misalignment between images and their corresponding optical flow. Even after warping images towards events, the problem persists.

Please take a look at the following picture to have a better understanding of the issue: videoframe_7248 Left : Optical flow (white) on events Middle : Events on image Right : Optical flow on image Note that the images are warped toward events and that the events here are not rectified.

Is there a way to have the optical flow perfectly (or almost perfectly) aligned with the images ?

Looking forward for your reply.

Thank you in advance !

magehrig commented 1 month ago

the optical flow gt is on the rectified event camera (see https://dsec.ifi.uzh.ch/data-format/)

AbdessamadElK commented 1 month ago

Hello Mr Mathias,

Thank you very much for your answer! So does that mean that there is no way for the images to be aligned with their corresponding optical flow ?

Yaozhuwa commented 1 month ago

@AbdessamadElK

So does that mean that there is no way for the images to be aligned with their corresponding optical flow ?

25 may help.

Pasted image 20241010202836

AbdessamadElK commented 4 weeks ago

@Yaozhuwa Thank you very much for your comment.

I already followed the steps in #25 to map the left image to the event camera frame, and also tried using the DSEC-det dataset directly. This allowed the images to be aligned with the events perfectly, but I am still struggling with the optical flow: videoframe_48

As you can see, the optical flow is in the rectified event camera as Mr. @magehrig indicated. You can also see that the warped images are indeed aligned with the distorted events but not with the optical flow.

Looking at the picture in your comment, It seems like you managed to align the optical flow with the images. Could you please tell me what kind of warping did you perform?

Yaozhuwa commented 4 weeks ago

@AbdessamadElK

Could you please tell me what kind of warping did you perform?

I did nothing to optical flow.

It seems you use the wrong optical flow to align with the image. For input image1 and image2, the optical flow will align with image1 rather than image2.

AbdessamadElK commented 4 weeks ago

@Yaozhuwa You might be right, I just loaded the flows and their corresponding images based on the file names, and I don't know if that's the right way to do it.

I will check again and see if that's the problem, thank you for your help!