Settings to reproduce result

ybgdgh / L3MVN

Leveraging Large Language Models for Visual Target Navigation

https://sites.google.com/view/l3mvn

62 stars 13 forks source link

Settings to reproduce result #12

Open edwardjjj opened 7 months ago

edwardjjj commented 7 months ago

Dear Authors,

I tried to reproduce your results by running the code with the following command

python main_llm_vis.py --split val --eval 1 --auto_gpu_config 0 \
-n 8 --num_eval_episodes 250 --load pretrained_models/llm_model.pt \
--use_gtsem 0 --num_local_steps 10

The success rate was only 0.522. Could you please share your settings to produce the same result in the paper? Thank you very much!

yusirhhh commented 4 months ago

@edwardjjj @ybgdgh Have you resolved this issue or trained the model? The results I obtained using the released model are as follows:

succ/spl/dtg: 0.529/0.246/4.094(2000),

edwardjjj commented 4 months ago

@edwardjjj @ybgdgh Have you resolved this issue or trained the model? The results I obtained using the released model are as follows:
succ/spl/dtg: 0.529/0.246/4.094(2000),

Hi, I have ran tests many times and never get the claimed results. Best was 0.52x

2003jiahang commented 2 months ago

@yusirhhh @edwardjjj I want to know how long have you run the whole dataset. I saw author run the model on the HM3D val,the whole dataset seem more than 2000 tasks？

edwardjjj commented 2 months ago

We ran several eval runs, each with 2000 episodes. The best result we got was 52.4% with feed forward method.

2003jiahang commented 1 month ago

@edwardjjj As far as I konw,the HM3D-val seem have more than 2000 tasks. Do I have misunderstanding? By the way ,I also can not reproduce the result.

xuai05 commented 1 month ago

How do you run the program code correctly? I encountered the following problems during the operation. However, my habitat-lab can run normally. Do you encounter the following problems? @edwardjjj

edwardjjj commented 1 month ago

How do you run the program code correctly? I encountered the following problems during the operation.

However, my habitat-lab can run normally. Do you encounter the following problems?

@edwardjjj

Hi, I think that's a habitat-lab version issue. You need to install the correct version.

xuai05 commented 1 month ago

I installed it from habitat-lab official website, url=https://github.com/facebookresearch/habitat-lab, can you tell me what version you are using? @edwardjjj My version is as follows:

edwardjjj commented 1 month ago

I installed it from habitat-lab official website, url=https://github.com/facebookresearch/habitat-lab, can you tell me what version you are using?

@edwardjjj

My version is as follows:

I think it was habitat-challenge-2022, check the tags in the habitat-lab repo.