How to extract event data information in the form of (x,y,t,p) from the output .aedat file?

[Rishabh-Samra]

I followed the instructions as mentioned to get event data information in the form of tuple (x,y,t,p) from input sequence of images. Even if I'm getting .aedat file in output folder, I am not getting how to read and extract event data info(in the form of {x,y,t,p} from this. I was checking this: https://pypi.org/project/aedat/ , and run "cp target/release/libaedat.so aedat.so" but it says no folder target. Still not sure how to deal with .aedat file to extract required event data information.

Also first the input frames are interpolated and then event data information is obtained from them. But I need not do the frame interpolation and directly want to get event frame information from corresponding video image frame sequence captured with 25 fps. How this can be done?

I request @duguyue100 and other people to help me address the above issues.

[duguyue100]

Hi, @Rishabh-Samra, v2e implements AEDAT 2.0 because we intend to use jAER (https://github.com/SensorsINI/jaer) to read the output file.

The project you mentioned implements an AEDAT 4 format, which is primarily used by the new platform called DV (https://inivation.com/dvp/). Therefore, there is no hope to read v2e output using this project.

You can read an AEDAT file in various ways, one of the solutions is dv-python, check the code here: https://gitlab.com/inivation/dv/dv-python/-/blob/master/dv/LegacyAedatFile.py

For your last question, you can use the --dvs_vid and --dvs_exposure options to output DVS videos. For example, you can set the exposure to 40ms, then the output video should be 25fps.

However, this is primarily used as a visualization tool instead of a data preparation tool. To properly generate the so-called "event frame", I suggest that you write your own custom function to bind events into frames, you can take a look at this library for more event representations: https://github.com/neuromorphs/tonic

After all, v2e is an event synthesis toolbox, it will not support post-processing routines.

[tobidelbruck]

You can also output the txt format aedat output. That is readable by any parser, but the files will be HUGE. There is a "format" of a sort, established by RPG in our dataset paper. Try it out.

On 1/3/2022 4:00 PM, Rishabh wrote:

I followed the instructions as mentioned to get event data information in the form of tuple (x,y,t,p) from input sequence of images. Even if I'm getting .aedat file in output folder, I am not getting how to read and extract event data info(in the form of {x,y,t,p} from this. I was checking this: https://pypi.org/project/aedat/ , and run "cp target/release/libaedat.so aedat.so" but it says no folder target. Still not sure how to deal with .aedat file to extract required event data information.

Also first the input frames are interpolated and then event data information is obtained from them. But I need not do the frame interpolation and directly want to get event frame information from corresponding video image frame sequence captured with 25 fps. How this can be done?

I request @duguyue100 https://github.com/duguyue100 and other people to help me address the above issues.

— Reply to this email directly, view it on GitHub https://github.com/SensorsINI/v2e/issues/29, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACMYQMSLJUKA32K43OEMU6DUUG22NANCNFSM5LFMXJMQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/SensorsINI/v2e/issues/29", "url": "https://github.com/SensorsINI/v2e/issues/29", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- Tobi Delbruck, Sensors Group:http://sensors.ini.uzh.ch Inst. for Neuroinformatics, UZH-ETH Zurich:http://www.ini.uzh.ch Building 55, Room G84, Winterthurerstr. 190, CH-8057 Zurich, Switzerland Tel lab +41 44 635 3038, Swiss cell +41 76 629 1500 (whatsapp), USA cell +1 626 510 2646, skype tobidelbruck

[tobidelbruck]

|Output: DVS events: --dvs_text DVS_TEXT Output DVS events as text file with one event per line [timestamp (float s), x, y, polarity (0,1)].|

[tobidelbruck]

It would be great to have AEDAT-4 output from v2e... probably quite easy to write it at least only for DVS brightness change events, which is all that v2e currently outputs. But frames would be very easy to add to since they are represented internal to v2e to make events.

The standard is here: https://inivation.github.io/inivation-docs/Software%20user%20guides/AEDAT_file_formats.html#aedat-40

[Rishabh-Samra]

Thanks @duguyue100 and @tobidelbruck for your feedback in the above raised issue. I still have doubts while implementing it for my case where input videos were captured at 24 or 25 fps. So I am taking a few consecutive frames(Say 6 frames) from a sequence of images(800 number of images) corresponding to a 25 fps video. And I want to generate my event data corresponding to those 6 frames only for now. So as suggested by @duguyue100 , when I tried with exposure duration of 40 ms keeping default values of all other arguments, I got the timestamp information corresponding to 148 frames in dvs-video-frame_times.txt but I need to have timestamp information corresponding to only those 6 frames of input videos, using which I made an input .mov file using opencv function, cv2.videowriter and setting fps=1. Is it incorrect to set fps=1? Otherwise this video of 6 frames will be of 0.25 sec. if we set fps=25.

So why output is giving timestamp information for 148 frames and how I can get the timestamp information corresponding to those 6 input frames only? How are arguments like timestamp resolution, dvs_exposure duration related to the calculation of getting accurate timestamps for 6 images?

[duguyue100]

@Rishabh-Samra, you get 148 frames instead of 6 frames in the output video perhaps because you left the --auto_timestamp_resolution on (which is the default value). If you don't need the fine temporal resolution, make sure you set this parameter to False.

In addition, if you supply the video as a sequence of images, you should also specify the video frame rate via --input_frame_rate, in this case, 25.

It seems to me that you want to get finer event timestamps while keeping the fps of the intensity frames. We haven't supported this mode naturally in v2e.

I believe there is a workaround, which is calculating the frame timestamps yourself. This is the way

frame_times  = np.array(range(n))  # n is number of frames in the original video.
frame_resolution = 1/frame_rate  # frame_rate = 25
frame_times = frame_times*frame_resolution

The key is that no matter how many times the video is upsampled, the time duration of both the original video and the upsampled is always the same.

You can validate it by first simulating events from v2e, and then visualizing the frame and events using a custom program such as OpenCV.

[Rishabh-Samra]

@duguyue100 Thanks alot for your above suggestions. But I still have doubts.

So I had already set the --auto_timestamp_resolution= False which was set in repo while running v2e.py script. Apart from that, I had used default parameter values like timestamp_resolution=0.003, --pos_thres=.15, --neg_thres=.15, --sigma_thres=0.03, and as you told I had kept dvs_exposure duration=0.040 sec(40 ms) as against 0.003 sec which was mentioned in repo. I was a bit confused as to what to specify for --stop_time because if I specify none as mentioned in the explanation, it is giving me an error. As you said, I will specify input_frame_Rate parameter to be 25. But if I will be doing that, then I have to give as an input only 0.25 seconds input video because while generating video I should make constant fps to be 25 then for 6 frames it will be 0.25 sec video only. So altering the fps to be 1 fps for making 6 sec input video will be incorrect if source video is captured at 25 fps? Or can this be done and I can specify input_frame_rate=1?

For generating close to the real event data for my case, can you suggest how I can modify these parameters that can be supported by v2e? Timestamp_resolution value can be changed to smaller values for my case and if yes, any suggestions for ideal range? Also modifying the input video fps will be helpful for getting better event data? What value should I keep for the same?

So for writing values of frame_times corresponding to 6 frames we will be having times to be (0, 1/25 , 2/25 , 3/25, 4/25, 5/25). So these values are of units 10^-6 sec and is this what the unit of time mentioned in output of dvs-video-frame_times.txt ?

[tobidelbruck]

Is this issue resolved now?

[duguyue100]

I think so.