How to run sub-task evaluation on the hold-out macro variation?

[yimengli46]

Hi @Jiayuan-Gu, thanks for sharing the nice implementation. I notice that we can run evaluations on 64 training scenes. (correct me if I'm wrong) Can you enlighten me on how to modify the config files to run evaluation on 20 test scenes and the macro variation scene? Do I need to self-generate the episodes?

[Jiayuan-Gu]

https://github.com/Jiayuan-Gu/hab-mobile-manipulation/blob/71b90da38a080ccd386c2c0f93cbf94fb745c739/configs/rearrange/skills/__base__.py#L52

For evaluation, passing EVAL.SPLIT [val,eval] as additional command line options can change the split to evaluate. The default split is "val". Unfortunately, the Habitat team held out the macro variation used for the eval split (cross-layout setting in the paper) this summer in order to hold the rearrangement challenge, which was public before. Thus, only "val" split is available in this public release.

https://github.com/Jiayuan-Gu/hab-mobile-manipulation/blob/71b90da38a080ccd386c2c0f93cbf94fb745c739/mobile_manipulation/eval_composite.py#L137

In general, you need to modify config.TASK_CONFIG.DATASET to set custom dataset.

Please follow instructions to download or generate your own episodes.

[yimengli46]

That solved my confusion. Thanks!

BTW, is the navigation skill trained on all the receptacles?

[Jiayuan-Gu]

I separately trained different navigation skills for 3 HAB tasks: TidyHouse (all receptacles except fridge, cabinet), PrepareGroceries (cabinet, fridge), SetTable (cabinet, fridge, table).

[yimengli46]

Thank you for the response. That clears my confusion.