Experimental design

[chcorbato]

(from https://github.com/rosin-project/metacontrol_sim/issues/19) The experimental design is being designed here: https://docs.google.com/document/d/1FRyrRr9ETblZj5e3-rSsoi8XIl6d_fzOLu6jlB4bxLY/edit#

and here we discuss the potential system configurations: https://docs.google.com/spreadsheets/d/12aB35Jj4qmarIq3OWHocATmnDfUa6ftp_j09W5twy1g/edit#gid=1491795427

@darkobozhinoski please add relevant updates to the issue so that we can all keep track

[chcorbato]

@darkobozhinoski once you have updated the experimental design on the overleaf and we agree on the experiments, then please close it

[chcorbato]

To my understanding we need to:

• [ ] Define evaluation metrics
• [ ] Define the test parameters and their values

[darkobozhinoski]

To my understanding we need to:

• [ ] Define evaluation metrics
• [ ] Define the test parameters and their values

I updated the test parameters with the perturbations that we plan to consider.You can find them in the following link: https://docs.google.com/document/d/1FRyrRr9ETblZj5e3-rSsoi8XIl6d_fzOLu6jlB4bxLY/edit#bookmark=id.l3f3e6f9ynl5

[chcorbato]

I updated the test parameters with the perturbations that we plan to consider.You can find them in the following link: https://docs.google.com/document/d/1FRyrRr9ETblZj5e3-rSsoi8XIl6d_fzOLu6jlB4bxLY/edit#bookmark=id.l3f3e6f9ynl5

Good! Only two comments (also in the gdoc):

• I propose not have the initial config as a test parameter. The system could always be initialized with the standard configuration (the one that performs better in the nominal task scenario) There is always such configuration, it is the one developers deploy.
• not sure about the calculation of the total number of cases: In my view, we have 3 factors corresponding to the 3 perturbations, and for each one we have 7 options (see number of levels of perturbation considering all types), so 7^3 cases of perturbations. In total then: 3*3*3*7*7*7 = 9261 Actually, thinking of it now again, this corresponds to cases having the 3 perturbations. We should add the cases having only 2 (7*7) and 1 (7) Total: 3*3*3*(7*7*7+7*7+7) = 10773

[chcorbato]

• I propose not have the initial config as a test parameter. The system could always be initialized with the standard configuration (the one that performs better in the nominal task scenario) There is always such configuration, it is the one developers deploy.

Ok, Darko convinced me that instead of multiple benchmarks/baselines, one per config. We consider only one but add as test param the initial config

[chcorbato]

@darkobozhinoski I propose you can close this issue once the experimental design section is complete in the paper (a complete draft)

[darkobozhinoski]

@chcorbato , I am somehow done (I might need to add one paragraph on component failures). This depends on what we decide today.

[chcorbato]

I think it is good for the moment, let's close it, since we have a complete Experimental design

Adding the Component failure case can be a separate issue (extra)