search

daniilidis-group / m3ed

M3ED Dataset
39 stars 3 forks source link

some confusion about depth and calib #1

Closed HTLeoo closed 1 year ago

HTLeoo commented 1 year ago

Thanks for the great work!

  1. I notice that the depth gt is not aligned with ovc images, right? It seems that ovc is 25Hz(40ms), but depth seems irregular.
  2. I notice that you offer many calibration files in h5 files. E.g., Cn_T_C0', 'Ln_T_L0' in depth_gt.h5, calib of RGB data, left/right grayscale camera data. Is it convenient for you to provide a simple chart to explain some of them?
fcladera commented 1 year ago

Hi @HTLeoo!

Depth is captured at LiDAR frame (10Hz) and it is not necessarily aligned with the ovc images, although it is time synchronized.

We are writing some documentation on the different file structures, and finishing some improvements for better readability. We hope to have this finished by the end of the week/early next week. We'll keep you posted here.

HTLeoo commented 1 year ago

Thanks for your quick reply. I still have some confusion about the 1st answer. Using 'car_urban_night_city_hall' sequence as an example, the timestamps of ovc images are [0 40000 80000 120000 ...], and the timestamps of depth_gt are [ 81938. 181917. 281800. 381722. ...]. If I want to align the depth with images, is it reasonable that the timestamps of depth_gt are approximately equal to [ 80000 180000 280000 380000 ...] since only 1-2 milliseconds are ignored? By the way, I believe that your work aims to promote multi-modality fusion (event+RGB/grayscale, event+Lidar), is it possible to provide a script to generate rectified coordinate frames (align ovc images with event pixel coordinates)?

fcladera commented 1 year ago

Hi @HTLeoo,

I am following-up after publishing v1.1.

As I mentioned, depth and images are not aligned, even if they are synchronized. The timestamp reported in the depth is the start of the swipe. There are two options:

  1. Pick the closest depth frame to the current image timestamp.

  2. Interpolate the poses and corresponding depth to the timestamp you need.

    Option 1 is the simplest one, but it may not provide the best results. We may implement 2 in a future version of the dataset, but it is not in our timeline now.

    Regarding the rectification script, you can find a similar script in the rectification we do with internimage.py

    https://github.com/daniilidis-group/m3ed/blob/main/build_system/semantics/internimage.py

    We are working on a set of sample scripts to use with M3ED.