search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
360 stars 77 forks source link

Questions about numbers in the paper #68

Closed Gasoonjia closed 3 years ago

Gasoonjia commented 3 years ago

Thank you guys for such outstanding dataset.

I have two questions about some numbers shown in the paper. First is about task type. In the paper you mention there are ~21,000 task in the training dataset (Figure F1) and each type would have roughly 3,000 task. Is the task type mentioned here traj_data.json['task_type]? If true, there are only ~6,500 traj_data.json files in train dataset, and there is only one task_type attribute in each json file. Therefore there are only ~6,500 task_type attribute in training dataset, which is a mismatch to the total task number. I was wondering if there is something wrong with my comparison.

Second, the paper shows that there are 2,685 combinations of task parameters (page 3). I suppose that each combination corresponding to each folder under different split (e.g. trainging set, val seen set, etc). However, both the number of folders in each data set and the total number of folders are inconsistent with this 2,685. So I wanna ask if there is something wrong with my understanding.

Thank you very much.

MohitShridhar commented 3 years ago

@Gasoonjia ALFRED contains 25K language annotations. There are ~8.5 trajectories in total, and for each trajectory (traj_data.json) we provided 3 or more annotations (in ['turk_annotations']['anns']), so ~8.5K x 3 = ~25K.

Gasoonjia commented 3 years ago

@MohitShridhar Thank you for your reply. However, I'm still confused about how to find the number of tasks (~21,000, Figure F1) and number of parameters combination (2,685 in page 3) in the dataset.

MohitShridhar commented 3 years ago

@Gasoonjia there are 2,685 task tuples <task, object, receptacle, scene>. For each task tuple, we sample 3 instances, so 2,685 x 3 = 8,055. Each task tuple can have lots of variations, e.g for the tuple <pick_and_place, apple, table, Room213> – instances can have different colored apples located in different parts of the room, the tables could be on a different side of the room.

MohitShridhar commented 3 years ago

Closing due to inactivity.