search

jacobkrantz / VLN-CE

Vision-and-Language Navigation in Continuous Environments using Habitat
https://jacobkrantz.github.io/vlnce/
MIT License
274 stars 54 forks source link

[RxR-Habitat] Using pose_trace data in RxR #13

Open awesomericky opened 3 years ago

awesomericky commented 3 years ago

Hello,

In RxR dataset (which is based on graph environment), they provide 'pose_trace' data which includes 'text_mask' that points out what words the agent saw in specific viewpoint. I am curious whether the 'instruction_id' in RxR exactly matches with 'instruction_id' in RxR-CE, which enable us to use the 'pose_trace' data based on your RxR-CE dataset

jacobkrantz commented 3 years ago

The instruction_id in RxR-Habitat matches to the instruction_id in RxR, so linking a pose trace to an RxR-Habitat episode should be as easy as that for Guide annotations. For Follower annotations, the pose traces can be matched using demonstration_id which matches to the RxR demonstration_id.

Its worth noting that the poses in the pose matrices do not map perfectly to the continuous environment node locations in the reference_path. This is due to adjusting the locations of some nodes while porting the dataset so they can be navigated to in the continuous environment.