Evaluation Procedure

[RudolfReiter]

We need a script or a workflow to take one of the configurations

• RL + environment
• MPC + enviornment
• safety filter + environment

with options

• random wall / fixed wall
• sampling close to wall / sampling randomly in feasible region

and collect the following data

• computation time of algorithm
• constraint violation of wall
• time to reach epsilon region
• closed loop cost as defined in MPC (Q,R)

[Erfi]

By computation time of algorithm are we referring to the "inference time" i.e. the time it takes the algorithms to give us the next action?

[Erfi]

Do we want to add random walls? If so I revise my comment on #31 and say that we should add the wall parameters also to the observation space.

[RudolfReiter]

Yes, inference time. I had problems with getting rid of all the pytorch specific things and really just count the time for matrix multiplications. Maybe you have a good idea for how to get the best estimate for embedded computation @Erfi ?

First, I would suggest to keep the wall constant. But have it already accordingly in the environment.

[Erfi]

Oki I can just add a timing for the predict function of the policies and perhaps get an average of it. Will make an issue for this.