search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
360 stars 77 forks source link

max_fails argument for leaderboard.py #83

Closed chanhee-luke closed 3 years ago

chanhee-luke commented 3 years ago

Hello,

I saw in the leaderboard.py script that there is a max_fails argument set to 10. Is this argument used to prevent models from exploiting information of action failing? If it is, I wonder why the value 10 was chosen. Thank you in advance for your answer!

MohitShridhar commented 3 years ago

@chanhee-luke, yes, max_fails is a limit that prevents agents from exhaustively trying out all actions by exploiting the simulator. For example during navigation, agents could bump along walls and corners instead of actually learning to navigate. We chose 10 to keep the limit as low as possible while still allowing for some unforeseen failures like collisions.