SL: Training Models and SL specific objects

[laxmikantbaheti]

Description/Motivation The supervised learning sub-package needs to be developed in order to offer a standardized processes for training Adaptive Functions. The brief objectives of the development are as following:

Task list

• [x] 1. Training Models:
  • [x] 1.1 SLScenario:
    • [x] A new class in sl/models_train.py
    • [x] Takes care of running one cycle, adaptation and evaluation, of adaptive function.
    • [x] This scenario also provides facility to store and save data and generate plots, independent from training (for inference purposes). See howto_sl_afct_100.
  • [x] 1.2 SLTraining:
    • [x] A new class in sl/models_train.py
    • [x] Trains the model, evaluate and test the model as per the required frequency. Evaluation and testing are two dfferent concepts here with similar process, but different data resources, since in some cases, the evaluation data changes over the trainin period, testing data remains the same.
    • [x] Calculates scores, and maintains highscore, for hyperparameter tuning.
  • [x] 1.3 SLTrainingResults
    • [x] Saves the results for SLTraining for all periods.
    • [x] Also provides possibility to automatically plot the results at the end of the training.
• [x] 2. Additional Objects
  • [x] #783
    • [x] The dataset is but a template class for an Offline Data resource.
    • [x] Responsible, to preprocess, and deliver data to a scenario.
    • [x] A detailed class diagram explains all the functions
    • Currently placed in SL, however, later to be moved to Application Support.
    • Currently no implementation of data provider template class. Shall be a topic in near future. Currently only supports a Dataset class and a custom dataset model SASDataset, which is still a model class. (Dataset providers are needed when we work with pool objects of datasets e.g. PytorchDatasets, MLPro Datasets, etc.)
  • [x] 2.2 Evaluation
    • [x] A new module sl/models_evals.py
    • [x] A new template class Metric(Log) and MetricValure(Element)
    • [x] Metric calculates the current evaluation metrics assigned to a model, as data and model are provided.
    • [x] The Metric object also stores scores and highscores of the model, for one instance.
    • [x] MetricValue stores the value calculated by the Metric.
  • [x] 2.3 SLDataStoring
    • [x] Similar to RLDatastorin, with a different frame variable "Epoch".

Additional Annotations

• A new object SLDataPlotting is created with a new plotting type, however, this can be later moved to lower level.
• A new element BatchElement is created which is just a class inherited from Element and no additional features. This just makes it easy to identify if the data is a batch or a single Element in case of online training.

Further topics

• [ ] Test scripts for SL objects

Related issues

• 434

Cross references ...

[laxmikantbaheti]

Hi @detlefarend , @steveyuwono , @rizkydiprasetya : This is the current issue for SL.

@steveyuwono , could you maybe do a review when possible? So maybe we can discuss next steps.