search

apple / ARKitScenes

This repo accompanies the research paper, ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data and contains the data, scripts to visualize and process assets, and training code described in our paper.
Other
662 stars 58 forks source link

How should I synchronize the .mov video and low-resolution depthmaps? #25

Closed EchoTHChen closed 2 years ago

EchoTHChen commented 2 years ago

I'm trying to download the datasets from RAW splits of ARKitScenes(the views of images in depth upsampling split is too sparse for me). I find there are only .MOV video and low-resolution depthmaps in most of scenes of RAW. So I want to synchronize the video and low-resolution depthmaps. How should I make it? How the timestamps of depthmaps and .mov videos are synchronized?

arik1089 commented 2 years ago

Hi @EchoTHChen,

Could you please elaborate on your use-case and requirements? If you just want to get color + depth frames at 60 FPS you can perhaps try the lowres_wide+lowres_depth assets, or we also have the vga_wide asset available in 30 FPS.

As for syncing the mov timestamps - You can find the timestamps of the mov frames inside the AVCaptureDeviceTypeBuiltInWideAngleCamera.1 Frame Summary mov track, under key OriginalTimestampWhenWrittenToFile - These are the timestamps we used in all the file names to sync the different assets.

Thanks, Arik.

EchoTHChen commented 2 years ago

Thanks. I think lowres_wide+lowres_depth is okay for me.

hugoycj commented 1 year ago

Hi @arik1089,

I am trying to extract the highres_wide at the same timestamp as the lowres_wide, I wonder how to get the timestamps of the mov frame using the key OriginalTimestampWhenWrittenToFile.

I have tried to use av libarary to check the metadata of .mov file, and get below information: { "streams": [ { "index": 0, "codec_name": "hevc", "codec_long_name": "H.265 / HEVC (High Efficiency Video Coding)", "profile": "Main", "codec_type": "video", "codec_tag_string": "hvc1", "codec_tag": "0x31637668", "width": 1920, "height": 1440, "coded_width": 1920, "coded_height": 1440, "closed_captions": 0, "film_grain": 0, "has_b_frames": 2, "pix_fmt": "yuvj420p", "level": 153, "color_range": "pc", "color_space": "smpte170m", "color_transfer": "bt709", "color_primaries": "bt709", "chroma_location": "left", "refs": 1, "id": "0x1", "r_frame_rate": "60/1", "avg_frame_rate": "28438200/473773", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895068, "duration": "112.801667", "bit_rate": "39831360", "nb_frames": "6771", "extradata_size": 108, "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Video", "vendor_id": "[0][0][0][0]", "encoder": "HEVC" }, "side_data_list": [ { "side_data_type": "Display Matrix", "displaymatrix": "\n00000000: -65536 0 0\n00000001: 0 -65536 0\n00000002: 0 0 1073741824\n", "rotation": -180 } ] }, { "index": 1, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x2", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895068, "duration": "112.801667", "bit_rate": "245242", "nb_frames": "6771", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 2, "codec_name": "hevc", "codec_long_name": "H.265 / HEVC (High Efficiency Video Coding)", "profile": "Main", "codec_type": "video", "codec_tag_string": "hvc1", "codec_tag": "0x31637668", "width": 640, "height": 480, "coded_width": 640, "coded_height": 480, "closed_captions": 0, "film_grain": 0, "has_b_frames": 2, "pix_fmt": "yuvj420p", "level": 93, "color_range": "pc", "color_space": "smpte170m", "color_transfer": "bt709", "color_primaries": "bt709", "chroma_location": "left", "refs": 1, "id": "0x3", "r_frame_rate": "10/1", "avg_frame_rate": "676800/67679", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1894956, "duration": "112.795000", "bit_rate": "4427301", "nb_frames": "1128", "extradata_size": 107, "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Video", "vendor_id": "[0][0][0][0]", "encoder": "HEVC" }, "side_data_list": [ { "side_data_type": "Display Matrix", "displaymatrix": "\n00000000: -65536 0 0\n00000001: 0 -65536 0\n00000002: 0 0 1073741824\n", "rotation": -180 } ] }, { "index": 3, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x4", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1894956, "duration": "112.795000", "bit_rate": "41023", "nb_frames": "1128", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 4, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x5", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 280, "start_time": "0.016667", "duration_ts": 1894812, "duration": "112.786429", "bit_rate": "14130330", "nb_frames": "6770", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 5, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x6", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 280, "start_time": "0.016667", "duration_ts": 1894813, "duration": "112.786488", "bit_rate": "129169", "nb_frames": "1128", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 6, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x7", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 280, "start_time": "0.016667", "duration_ts": 1894812, "duration": "112.786429", "bit_rate": "573305", "nb_frames": "6698", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 7, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x8", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 280, "start_time": "0.016667", "duration_ts": 1894812, "duration": "112.786429", "bit_rate": "555566", "nb_frames": "6695", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 8, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0x9", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 280, "start_time": "0.016667", "duration_ts": 1894812, "duration": "112.786429", "bit_rate": "1061522", "nb_frames": "6770", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 9, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0xa", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895093, "duration": "112.803155", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 10, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0xb", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895093, "duration": "112.803155", "bit_rate": "654757", "nb_frames": "6770", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 11, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0xc", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895093, "duration": "112.803155", "bit_rate": "8538261", "nb_frames": "6218", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } }, { "index": 12, "codec_type": "data", "codec_tag_string": "mebx", "codec_tag": "0x7862656d", "id": "0xd", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/16800", "start_pts": 0, "start_time": "0.000000", "duration_ts": 1895093, "duration": "112.803155", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, "tags": { "creation_time": "2020-12-07T12:02:18.000000Z", "language": "und", "handler_name": "Core Media Metadata" } } ] }

I wonder whether AVCaptureDeviceTypeBuiltInWideAngleCamera is in one of the "Core Media Metadata" tracks. If so, how to decode it

arik1089 commented 1 year ago

Hi @hugoycj ,

The timestamps of the highres_wide and the lowres_wide frames are not synced, so you can't match them. They are both in 10 FPS and are generated from different post-processing tools, which were not time-synced.

As for accessing the mov timestamps - We do not provide code to access this metadata at this time, but you can use Apple Documentation, such as this to do it yourself. Please notice that the timestamps are inside the metadata of the wide camera track, and not the general metadata of the mov (which seems like what you printed here).

Thanks.