Navigation Policy Testing

[yihang-xdu]

Dear Authors,

When I test the model using "python eval.py --mode=test_DDN --eval_path=$path_to_saved_model$ --dataset_mode=$train,test$ --seen_instruction=$0,1$ --device=cuda:0 --epoch=500 --eval_ckpt=$idx$" , it accurs an error "FileNotFoundError: [Errno 2] No such file or directory: './pretrained_model/VG.pt'" .

I didn't find a file named 'VG.pt' . Can you upload it ? Thanks a lot !

[whcpumpkin]

Hi, Sorry, I forget to upload this model... I will upload it now. Please check the Materials Download later today.

[yihang-xdu]

OK , Thanks for your timely reply !

[yihang-xdu]

Dear Authors,

I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" :

ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64

I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

[whcpumpkin]

Dear Authors,

I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" :

ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64

I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

Hi, My AI2Thor version is 5.0.0. I do not know what's wrong with you. Maybe you can refer AI2Thor's official repo for help.

[whcpumpkin]

Dear Authors,

I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" :

ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64

I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

BTW, I recall that AI2thor will automatically download a resource pack when it first runs, which may require an Internet connection. If you are in mainland China, you may need to use a VPN.

[yihang-xdu]

Thanks for your explanation . I think this is really a server network problem . I will try in other ways . Thanks a lot !

[yihang-xdu]

Dear Authors,

I have obtained the test results in the "seen_scene" and "seen_instruction" cases. However, I'm not sure if the results fall within a normal range. Additionally, I noticed that the amount of data used was different for each epoch. Can you give me some advice?

[whcpumpkin]

Dear Authors,

I have obtained the test results in the "seen_scene" and "seen_instruction" cases. However, I'm not sure if the results fall within a normal range. Additionally, I noticed that the amount of data used was different for each epoch. Can you give me some advice?

Hi, Can you tell me how to select the tested model? With 200 instructions and 200 rooms in the seen_instruction, it is not unusual to see some fluctuations in the results when testing. The image below is a screenshot of my TensorBoard records during testing, and as you can see, the fluctuations are also quite significant.

Therefore, I think the results you got are normal.

I noticed that the amount of data used was different for each epoch

The data is loaded in the dataloader. In theory, this phenomenon should not occur. I hadn't noticed that

[whcpumpkin]

sele_seccess is very unstable. The baselines also use this VG.pt model to make the pipeline integrated. I recall the VG's success rate is 80% when the object is in the field of view during selecting the VG model.

I will release the VG model's training code. But the dataset to train the VG model was deleted... I think if possible, you can use any segmentation model (e.g., SAM) and LLM to replace the VG model.

[yihang-xdu]

Dear Authors,

Thanks for your reply ! I select the tested model with the highest accuracy on the validation set . The index 10 is high and smooth enough.

I will try in the future according to what you said. Thanks for your explanation again !

[whcpumpkin]

Dear Authors,

Thanks for your reply ! I select the tested model with the highest accuracy on the validation set . The index 10 is high and smooth enough.

I will try in the future according to what you said. Thanks for your explanation again !

Hi, your selection is right. It might be due to some random factors affecting the experimental results. I have also reported the standard deviation in the paper.

[yihang-xdu]

That's fine. Thanks a lot !

[gqsmmz]

Dear Authors,

I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" :

ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64

I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2：

I really want to know how you solved this problem. Looking forward to your reply!~

[whcpumpkin]

Dear Authors, I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" : ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64 I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2：

I really want to know how you solved this problem. Looking forward to your reply!~

Hi, What are your version of AI2Thor and your operating system? if you run in a headless machine, you should add xvfb-run -a at your beginning of your command, like xvfb-run -a python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2

[gqsmmz]

Dear Authors, I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" : ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64 I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2： I really want to know how you solved this problem. Looking forward to your reply!~

Hi, What are your version of AI2Thor and your operating system? if you run in a headless machine, you should add xvfb-run -a at your beginning of your command, like xvfb-run -a python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2

my ai2thor version is 5.0.0. my operating system is 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64), which is a headless machine, and is unable to connect to the external network.

I have tried to add xvfb-run -a, it appears the same problem and seems like not the problem of this.

Based on your discussion above, it is speculated that it may be necessary to connect to the external network. However, if it is not possible to connect to the external network, do you have any suggestions？

[whcpumpkin]

Dear Authors, I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" : ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64 I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2： I really want to know how you solved this problem. Looking forward to your reply!~

Hi, What are your version of AI2Thor and your operating system? if you run in a headless machine, you should add xvfb-run -a at your beginning of your command, like xvfb-run -a python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2

my ai2thor version is 5.0.0. my operating system is 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64), which is a headless machine, and is unable to connect to the external network.

I have tried to add xvfb-run -a, it appears the same problem and seems like not the problem of this.

Based on your discussion above, it is speculated that it may be necessary to connect to the external network. However, if it is not possible to connect to the external network, do you have any suggestions？

Hi, Maybe this issue can help you: https://github.com/allenai/ai2thor/issues/931

The first time you run AI2Thor, the controller will automatically download a resource pack. If you don't have an external network, you can try downloading this resource pack manually and put it to the right path (In my machine, it is ~/.aithor/releases)

Another way is that you can download the resource pack in a machine with network, and copy the resource pack to your running machine.

[gqsmmz]

Dear Authors, I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" : ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64 I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2： I really want to know how you solved this problem. Looking forward to your reply!~

Hi, What are your version of AI2Thor and your operating system? if you run in a headless machine, you should add xvfb-run -a at your beginning of your command, like xvfb-run -a python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2

my ai2thor version is 5.0.0. my operating system is 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64), which is a headless machine, and is unable to connect to the external network. I have tried to add xvfb-run -a, it appears the same problem and seems like not the problem of this. Based on your discussion above, it is speculated that it may be necessary to connect to the external network. However, if it is not possible to connect to the external network, do you have any suggestions？

Hi, Maybe this issue can help you: allenai/ai2thor#931

The first time you run AI2Thor, the controller will automatically download a resource pack. If you don't have an external network, you can try downloading this resource pack manually and put it to the right path (In my machine, it is ~/.aithor/releases)

Another way is that you can download the resource pack in a machine with network, and copy the resource pack to your running machine.

Thank you so much! I have try downloading this resource pack manually, put it to the right path and run the test code successfully. In the experiment of Seen Scene and Seen ins. , I found that after I choose the checkpoint whose validation pretrained accuracy was about 86%, the SSR is just about 2%,the NSR is about 19%, the NSPL is about 8%.

• I'd like to ask if the output of "sele_success_num" is SSR, "navi_success_num" is NSR, "spl" is NSPL?

• what's meanning of the output as "dis_goal"?

• What do the values in parentheses of the results in the first six rows and the last row of the table mean, because you did not label the ablation as Ours_w/o-uattr_transformer

• my "sele_success_num" is juat 2%, which I think maybe too low compared to your result in paper like 7.5%, do I understand right? And why am I so low? Do I need to make some improvements

[whcpumpkin]

Dear Authors, I met an error when I run "self.controller = Controller(scene=self.dataset[0], renderDepthImage=False, gridSize=0.05, snapToGrid=True, visibilityDistance=1.5, width=300, height=300,server_timeout=500)" : ValueError: Invalid commit_id: f0825767cd50d69f666c7f282e54abfe58f1e917 - no build exists for arch=Linux platforms=Linux64 I'm not sure if it's due to the version of AI2-THOR or something else. Can you give me some advice ?

In the test period, when I used the selected model to test the final performance, I found the AI2thor's problem(seems like the connecting problem?)，my code was python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2： I really want to know how you solved this problem. Looking forward to your reply!~

Hi, What are your version of AI2Thor and your operating system? if you run in a headless machine, you should add xvfb-run -a at your beginning of your command, like xvfb-run -a python eval.py --mode=test_DDN --eval_path=$path$ --dataset_mode=train --seen_instruction=1 --device=cuda:0 --epoch=500 --eval_ckpt=2

my ai2thor version is 5.0.0. my operating system is 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64), which is a headless machine, and is unable to connect to the external network. I have tried to add xvfb-run -a, it appears the same problem and seems like not the problem of this. Based on your discussion above, it is speculated that it may be necessary to connect to the external network. However, if it is not possible to connect to the external network, do you have any suggestions？

Hi, Maybe this issue can help you: allenai/ai2thor#931 The first time you run AI2Thor, the controller will automatically download a resource pack. If you don't have an external network, you can try downloading this resource pack manually and put it to the right path (In my machine, it is ~/.aithor/releases) Another way is that you can download the resource pack in a machine with network, and copy the resource pack to your running machine.

Thank you so much! I have try downloading this resource pack manually, put it to the right path and run the test code successfully. In the experiment of Seen Scene and Seen ins. , I found that after I choose the checkpoint whose validation pretrained accuracy was about 86%, the SSR is just about 2%,the NSR is about 19%, the NSPL is about 8%.

• I'd like to ask if the output of "sele_success_num" is SSR, "navi_success_num" is NSR, "spl" is NSPL?
• what's meanning of the output as "dis_goal"?
• What do the values in parentheses of the results in the first six rows and the last row of the table mean, because you did not label the ablation as Ours_w/o-uattr_transformer
• my "sele_success_num" is juat 2%, which I think maybe too low compared to your result in paper like 7.5%, do I understand right? And why am I so low? Do I need to make some improvements

Hi, I am glad that you reproduced my results.

I'd like to ask if the output of "sele_success_num" is SSR, "navi_success_num" is NSR, "spl" is NSPL?

Yes.

what's meanning of the output as "dis_goal"?

dis_goal is an unused metric. It records the average distance from the robot to the nearest target at the end of the episode.

What do the values in parentheses of the results in the first six rows and the last row of the table mean, because you did not label the ablation as Ours_w/o-uattr_transformer

The values in parentheses are the standard deviation under several different random SEEDs. This is the standard deviation only for algorithms that require training (including the first 6 rows as well as the ablation experiments). The algorithms in rows 7-12 do not require training, so there is no standard deviation reported.

my "sele_success_num" is juat 2%, which I think maybe too low compared to your result in paper like 7.5%, do I understand right? And why am I so low? Do I need to make some improvements

VG.pt is used for the final step of detection in all experiments (including baselines and main experiments). Its results are very unstable . I'm not quite sure why your nsr is close to 20% but your ssr is low (close to random). Some better designs might consider using LLM combined with Yolo for picking bounding boxes.