search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
360 stars 77 forks source link

pddl_params ony avaialble for train data? #60

Closed wcarvalho closed 3 years ago

wcarvalho commented 3 years ago

How come pddl_params are only available for the train data? Is there a way to know the task "item" + "receptacle" are for validation/test?

wcarvalho commented 3 years ago

Even if we don't use it for training, this information seems useful for analysis

MohitShridhar commented 3 years ago

@wcarvalho we don't provide metadata for test scenes on purpose. This is a standard practice is most datasets/benchmarks to prevent users from exploiting the metadata instead of actually solving the challenge. ALFRED is a vision-language challenge where we ultimately want to train agents that purely operate with ego-centric vision and language instructions as input. So the test scenes are specifically reserved for assessing this.

However, for internal analysis, you can use the validation sets (val_seen an val_unseen), which should contain pddl_params etc.

wcarvalho commented 3 years ago

thanks - that makes sense!