search

robot-learning-freiburg / MM-DistillNet

PyTorch code for training MM-DistillNet for multimodal knowledge distillation. http://rl.uni-freiburg.de/research/multimodal-distill
GNU General Public License v3.0
59 stars 14 forks source link

Question about the dataset #9

Open drydenwiebe opened 3 years ago

drydenwiebe commented 3 years ago

Hello,

Thanks for putting out this great dataset!

I just have a question about the rgb and audio modalities of the dataset. When reading the README.txt it seems that there should be 8 audio files per rgb file (for the drive/timestep).

But when I unzip the audio files only, there are 1326187 (/8 = about 165772) audio files and when I unzip the rgb files there are 175344 files. This means that there are not enough corresponding audio files for rgb files.

Is this intended or did I get something wrong along the way?

Thanks again!

franchuterivera commented 3 years ago

Hello, thanks for your interest in our project.

If you are looking to use synchronized audio and RGB modalities, I recommend you to use the IDs provided in the train/val/test files as they are guaranteed to have all modalities available.

You are likely going to find more RGB images than audio images because even though the rgb and audio capture started at the same time, to ensure there is perfect synchronization, we omit the first and last seconds of each recording. We decided to keep all RGB files (even though there is not an audio modality) because there will be a depth or thermal file that can still be leveraged.

More specifically, we used a ROS bag file and a separate audio capture process which is aligned using a gps-clock-based synchronization. The first milliseconds might not be properly aligned, so we produce an audio file only when the synchronization is guaranteed, and that happens a few seconds later after the bag file started recording.

catherine-qian commented 2 years ago

Dear authors,

  1. I would like to verify with you that, for the downloaded dataset, the structure and size should be: 432K drive_day_2020_03_18_15_42_58 32M drive_day_2020_04_14_14_33_17 188M drive_day_2020_04_14_15_10_13 104M drive_day_2020_04_14_15_20_14 121M drive_day_2020_04_14_16_16_27 507M drive_day_2020_04_21_15_48_20 284M drive_day_2020_04_21_15_58_22 2.6G drive_day_2020_05_21_20_55_36 3.5G drive_night_2020_05_21_21_05_38 3.9G drive_night_2020_05_21_22_22_50 9.1G drive_night_2020_05_31_22_00_00 104M drive_night_2020_05_31_22_23_23

  2. i didn't find the aforementioned train/val/test partions. Could you please specify it?

Thanks in advance for your help!