search

apple / ARKitScenes

This repo accompanies the research paper, ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data and contains the data, scripts to visualize and process assets, and training code described in our paper.
Other
662 stars 58 forks source link

Question about camera orientation (portrait and landscape) #10

Closed garrickbrazil closed 2 years ago

garrickbrazil commented 2 years ago

I am trying to extract frames from the Raw dataset and running into trouble/confusion related to the orientation of the images which vary between Portrait and Landscape modes from video to video.

Here are a few questions I have on this topic.

  1. Is there any annotation or way of determining what the correct orientation is for the Raw images/annotations/intrinsic? Most seem to be rotated by -90 degrees but not all as far as I can tell. It seems that videos in Landscape mode are mostly not rotated but could occasionally be upside down in my tests.

  2. Are the Raw videos always in the "correct" orientation? They seem to be at a glance, so I have assumed this for now.

  3. Is it known whether the camera operators switch between Landscape and Portrait modes in the middle of a video? If it's not known, then was it an intention?

Thank you and sorry if this is covered somewhere in the code that I missed.

PeterZheFu commented 2 years ago

Hi @garrickbrazil, Thanks for your interest and reaching out~ For images in 3dod/ directory, I use .traj poses to infer the orientation.

garrickbrazil commented 2 years ago

Thank you for the fast response! I don't mind reasoning with the .traj files, that seems like a great solution in fact.

Can you elaborate on how the .traj poses imply the camera orientation? From my understanding it would be ambiguous whether the camera was lifted vertically or is rotated sideways and was moved horizontally instead. Am I missing a useful cue that is captured in the camera poses or the 3D object annotations? Any tips are appreciated.

As an aside, I notice that there is a file of metadata.csv which captures orientation for the upsampling split but I don't see it for Raw or 3DOD. In the csv, each video is labeled Up, Down, Left, or Right, which does seem to be what I'm looking for. Naturally, I'm wondering does the metadata exist for Raw or 3DOD?

PeterZheFu commented 2 years ago

Hi @garrickbrazil, For 3DOD task, we use camera poses in the .traj file to take care of the depth images (since we do not use the low_res color images in 3DOD). Currently we do not include the metadata (with Up, Down, Left, or Right) in 3DOD directory.

PeterZheFu commented 2 years ago

closing the issue for now. Feel free to re-open it if you have further comments. Thanks.

arik1089 commented 2 years ago

Hi @garrickbrazil, We have added metadata files for each of the datasets, which will be available to download after the latest commit to our download script (the metadata will be included automatically in every download). Please see the “sky_direction” field for the orientation data of each video. Keep in mind this only describes the sky direction of the video, while the images should always be in landscape resolution. For example: Images which are with “left” sky orientation should be rotated 90 degrees clockwise and were originally shot in portrait, while images which are with sky orientation “up” should not be rotated and were originally shot in landscape. Thanks again for your question.

louzq16 commented 2 years ago

There exist many wrong sky_direction labels in raw metadata ....

yifanjiang19 commented 2 years ago

"There exist many wrong sky_direction labels in raw metadata ...." same observation here