OA: Benchmarking Functionality

[steveyuwono]

Description/Motivation

MLPro-OA-Streams: Benchmark

Automates the benchmarking process over two or more scenarios. We are planning to introduce a basic function for benchmarking in the low level, which can be inherited for RL. #632 Benchmarking criteria? evaluation scores, training parameters, etc.?

Task list

• [ ] 1. New class OABenchmark
  • [ ] Stream provider and list of workflows to be benchmarked as parameters
  • [ ] Trains every workflow on all streams of the stream provider
  • [ ] Creates performance statistics
• [ ] 2. Suitable howtos
• [ ] 3. RTD documentation

Related issues

632 #718 #717

Cross references ...

[detlefarend]

To be discussed in one of our next oa meetings...

[syamrajsatheesh]

..