run initial experiments

[marioney]

As the items from https://github.com/rosin-project/metacontrol_sim/issues/26 are completed, I can now run the initial experiments (27 Runs: 3 initial points, 3 goals and 3 navigation configurations)

@darkobozhinoski @chcorbato The perturbations described in https://github.com/rosin-project/metacontrol_nav/issues/4 are not yet implemented, do we need them for the initial set of experiments?

@darkobozhinoski is the safety observer working? or should we just use the qa_energy?

[chcorbato]

If easy to implement, it would be very handy to:

• select if to run only one experiment or the complete batch
• for the 1 experiment only, if would be handy to select which perturbations to add (up to three) This would allow to test the metacontrol functionality, which is different from running experiments. Maybe I should create a separate issue @marioney ?

UPDATE: separate issue for Testing #32

[marioney]

* This would allow to test the metacontrol functionality, which is different from running experiments. Maybe I should create a separate issue @marioney ?

Yes, we should create a separate issue for this. Actually more than one as we need to define the perturbations (obstacle positions etc) and another for the changes you are suggesting

[darkobozhinoski]

As the items from #26 are completed, I can now run the initial experiments (27 Runs: 3 initial points, 3 goals and 3 navigation configurations)

@darkobozhinoski @chcorbato The perturbations described in rosin-project/metacontrol_nav#4 are not yet implemented, do we need them for the initial set of experiments?

@darkobozhinoski is the safety observer working? or should we just use the qa_energy?

@marioney , Both are implemented. I just pushed a new version that normalizes both "safety" and "energy" values. Please check all the topics in the safety observer: https://github.com/rosin-project/rosgraph_monitor/blob/observers/src/rosgraph_monitor/observers/safety_quality_observer.py For the safety observer, we mentioned that probably there is a possibility to take the acceleration (check the discussion in this issue. Let me know if that is ok?

@marioney , for the changes on perturbations check the Experimental Design document. It has been updated regarding the Test parameters: https://docs.google.com/document/d/1FRyrRr9ETblZj5e3-rSsoi8XIl6d_fzOLu6jlB4bxLY/edit#bookmark=id.l3f3e6f9ynl5

[darkobozhinoski]

* This would allow to test the metacontrol functionality, which is different from running experiments. Maybe I should create a separate issue @marioney ?

Yes, we should create a separate issue for this. Actually more than one as we need to define the perturbations (obstacle positions etc) and another for the changes you are suggesting

To define perturbations please check the document above. In general, this is how I see it:

• Number of perturbations: Choose from 1 to 3
• For each perturbation:
  • select [Type of perturbation, level of perturbation]

There are three types of perturbation:

1. Cluttered environment,
2. Unexpected increase in power consumption,
3. Component_failure

Levels of perturbation per type:

• Cluttered environment = {low, medium, high}
• Increase in power consumption = {10%, 20%, 30%}
• Component_failure ={laser_error}

More info in the document

[marioney]

Levels of perturbation per type:

* Cluttered environment = {low, medium, high}

We need to define what do we mean by low / medium or high cluttered.

I see two ways of doing this, have fixed size obstacles and add different number of them, or have a fixed number of obstacles and change their size.

[darkobozhinoski]

@marioney , that is defined in the document to some extend.

• Low means 1 obstacle;
• Medium means 2 obstacle with "large space" between them;
• High means 3 obstacles with "short" distance between them;

Short vs. large distance between obstacles is in order to play with the inflation radius.

The exact values will come from you. Regarding the size of the obstacles, this can be fixed (for now).

[chcorbato]

Addressed by the experiments script: https://raw.githubusercontent.com/rosin-project/metacontrol_experiments/master/run_batch_sim.sh