Create metrics for the exemplar

[Rezenders]

We need to have metrics to measure how well the managing system is performing. My idea so far is to measure the time the AUV takes to complete the mission

[Rezenders]

@JulianePa suggestions:

• We could try to find some metric that measures how indecisive a managing system is. Maybe measure how often it switches between configurations and divide that by the time it needs to complete the mission. Because i guess it's not good to always switch between configurations if it's not necessary.
• We could measure how long it takes the managing system to react to a fault of the managed system. E.g. the time from when a thruster broke to when the managing system adapted the managed system to only use 5 thrusters.
• We could also detect whether it adapts to a fault at all/whether it goes to a higher altitude if possible. This is probably related to the time it takes to complete the mission.
• We could measure the distance it covers until it completes the mission. Probably also related to the time, but it would show whether the robot moves around unnecessarily much.

[jeroenzwan]

Other options:

• Number of unique configurations used. The more configurations used could be both positive and negative however. Many configurations used can show high adaptability, but also indecisiveness.
• To add to the time needed to react to a fault in the managed system: the time needed to reason about adaptation and apply.
• Maybe not related to the simulation, but it might be interesting to show number of unique adaptation options that are available. This would show high adaptability.
• One that is definitely needed is to show how much faster/more accurate the UUV is with adaptation vs without.

[EGAlberts]

I would say we need a utility function. This utility function should be based on the requirements of the system. This paper has a nice example of deriving one: http://www.es.mdh.se/pdf_publications/4695.pdf

[Rezenders]

From the meeting:

Important metrics:

• Time for searching pipeline
• Time for inspecting the pipeline (minimize it)
• Sum both times
• Distance of the pipe detected and inspected (if we set a time limit) (maximize it)
• Relationship between them