Development of a trainings-strategy: Optimum-Strategy

[jamalissa]

Description:

This issue outlines the creation of a new training strategy "Optimum Strategy."

Details of the Strategy:

1. Goal:
   • The goal of the Strategy is to systematically find the best-performing model and training configuration for a specific task, optimizing the selected performance metric to achieve the highest performance for a given task.
2. Methodology:
   1. Conduct a train-test split to evaluate model performance accurately throughout the training sessions.
   2. Initially train all candidate models for a brief period (e.g., T₁= 5 minutes).
   3. Measure and record the accuracy of each model after T₁.
   4. Continue training the models for an additional period (T₂ = T₁ * 2).
   5. After T₂, compare the performance metrics (e.g., accuracy) with the t1 results.
   6. Repeat this process iteratively (Tₙ = Tₙ₋₁ * 2), until all of the selected models do not show improved performance.
   7. Capture and display the final runtime and performance metrics of the most optimum model to the user (The model with the top performance metric ).
   8. After determining the Optimum Model, re-train it using the entire dataset (without a train-test split) to maximize its performance.

Expected Behavior:

User Interaction:

• Users can select "Optimum Strategy" from the available training strategies.

Output Information:

• Post-training, users should receive details on:
  • The chosen optimum model and the associated library used.
  • Actual runtime for training on the selected dataset.