search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
360 stars 77 forks source link

Bug in evaluation of the GotoLoacation subgoal #59

Closed 594zyc closed 3 years ago

594zyc commented 3 years ago

While evaluating the GotoLocation subgoal, the completeness is determined by whether the agent is close enough and the target object of the next subgoal is seen in the last state of the GotoLocation subgoal (see here). However, if in the next subgoal the agent needs to open some nontransparent thing first (e.g. most commonly a fridge) and then pickup/slice/etc the object, the visibility check will fail and the GotoLocation subgoal will be incorrectly classified as unsuccess. To my knowledge this is a bug in the evaluation process. If I misunderstand anything please just ignore this issue. Thank you!

MohitShridhar commented 3 years ago

@594zyc this is a bug. The subgoal evaluation numbers for Goto might be slightly higher than reported. I'll take a look at this soon and add an errata. Thanks

MohitShridhar commented 3 years ago

Updated the success criteria for Goto and added an errata page.

MohitShridhar commented 3 years ago

Closing for now.