How to interpolate the annotation

[lixinghe1999]

Dear EGOEXO4D team,

Thanks for releasing the dataset and challenges. After reading the documentation and playing with the code, I still find some confusing points.

1. EgoPose is annotated with frame idx; how do we convert it to time, and then can we use the raw audio & IMU & Image? Should I assume the FPS = 30 or 60? Is it a fixed value?
2. I observed in many annotations that the frame idx is not starting from 0. Is it normal or not?

I totally agree that the baseline in this repo works since the input "camera pose" and output "skeleton" are both labeled frame-wise. I wonder how to use other modalities. I appreciate the clarification.

[juanitapuentes]

Hi!

The EgoPose annotations have two formats: manual and automatic. Manual annotations are recorded at 30 FPS, while automatic annotations are at 60 FPS. You can use other modalities and convert them into frames. For example, Aria device videos are at 30 FPS. When using images, remember that you can only use egocentric video.

Regarding your observation about the frame indices not starting from zero, this is normal. The annotations start when the key points are visible in at least two of the GoPro cameras, which is necessary to obtain a 3D body pose for a frame. Thus, the frame index does not always start at zero.

[suyogduttjain]

Hi,

A bit more clarification. The manual annotations were actually done 10 FPS i.e. we annotated every 3rd frame so intermediate 2 frames need to be interpolated if required. We release automatic annotations at 30fps. All our released take videos are at 30FPS. Regarding frame number, we did annotations by selecting different parts of the videos (usually a short 10 second clip within the entire video). The frame number refers to the exact location in the video where we did the annotations. You can assume 30FPS to get time locations or can also look at timesync.csv files which come with each take to map frame number to timestamps.