search

YicongHong / Recurrent-VLN-BERT

Code of the CVPR 2021 Oral paper: A Recurrent Vision-and-Language BERT for Navigation
Other
149 stars 27 forks source link

R2R Test Unseen #21

Closed jasscia18 closed 11 months ago

jasscia18 commented 11 months ago

Thank you so much for the great work you do. I reproduced Recurrent VLN on the R2R datasets according to the file readme.md. But I only get the results for the validation set, how can I get the results of the Test Unseen?

YicongHong commented 11 months ago

Hi @jasscia18, thanks for your interest in our work! The Test Unseen ground truths are reserved on R2R's test server for fair comparison, you need to upload the result .json file to the server to see the SR/SPL/etc. (please see https://eval.ai/web/challenges/challenge-page/97/evaluation).

To get the result .json file, I believe you only need to run the test_agent.bash and set --submit 1.

Cheers, Yicong

jasscia18 commented 11 months ago

Hi Yicong, I followed your instructions to run the test_agent. bash script with the --submit 1 flag to obtain the result .json. However, I encountered an error during the execution of the script.

1697095273034

I would greatly appreciate your guidance on how to address this error and successfully generate the result .json file. Any assistance you can provide would be extremely helpful in our research efforts.

Thank you in advance for your time and support.

YicongHong commented 11 months ago

Hi, since we don't have the ground-truth paths for the test set, we can't measure nDTW. You only need to comment out all lines for computing nDTW. And to save time, you can set the validation data to be only test unseen so that you don't need to re-eval for the other three splits every time.

Cheers