MapGPT

The official implementation of MapGPT. [Paper] [Project]

MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation.

Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, Kwan-Yee K. Wong.

Annual Meeting of the Association for Computational Linguistics (ACL 2024).

If you have any questions, please contact me by email: jqchen(at)cs.hku.hk

Setup

Install Matterport3D simulators: follow instructions here. We use the latest version instead of v0.1.

Install requirements:

conda create -n MapGPT python=3.10
conda activate MapGPT
pip install -r requirements.txt

Prepare data:

• You can follow DUET and set the annotations for testing val-unseen set.
• We sample a subset containing 72 scenes and 216 cases for quick and cost-effective testing. You can download the corresponding MapGPT_72_scenes_processed.json and place it in the datasets/R2R/annotations directory.
• The observation images need to be collected in advance from the simulator. You can use your own saved images or use the RGB_Observations.zip we have processed.

GPT key: please set your API key here.

Inference

In addition to the reported results of GPT-4v in the paper, we have also included the implementation of latest GPT-4o which is faster and cheaper.

You can run the following script where --llm is set as gpt-4o-2024-05-13 and --response_format is set as json.

bash scripts/gpt4o.sh

The performance comparison between two implementations on a sampled subset is as follows. GPT-4o can achieve better NE but slightly worse SR.

Note that you should modify the following part in gpt4o.sh to set the path to your observation images, the split you want to test, etc.

--root_dir ${DATA_ROOT}
--img_root /path/to/images
--split MapGPT_72_scenes_processed
--end 10  # the number of cases to be tested
--output_dir ${outdir}
--max_action_len 15
--save_pred
--stop_after 3
--llm gpt-4o-2024-05-13
--response_format json
--max_tokens 1000

Citation

@inproceedings{chen2024mapgpt,
  title={MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation},
  author={Chen, Jiaqi and Lin, Bingqian and Xu, Ran and Chai, Zhenhua and Liang, Xiaodan and Wong, Kwan-Yee~K.},
  booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics",
  year={2024}
}