Feature: (AutoML) Refactor Trainer for metrics to possibly be measured before training.

Neuraxio / Neuraxle

The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.

https://www.neuraxle.org/

Apache License 2.0

608 stars 62 forks source link

Feature: (AutoML) Refactor Trainer for metrics to possibly be measured before training. #459

Closed vincent-antaki closed 1 year ago

vincent-antaki commented 3 years ago

Right now, we only measure metrics after each epoch. It could be useful to measure metrics before the first epoch as to gain better insight on the training process.

Considerations:

not every pipeline are able to transform before performing a fit. We therefore need to be able to tell the responsible instance (AutoML, Trainer or MetricCallback) whether to perform it or not
we might want the ability to perform this for some metrics and not for others.
the TrialMetricPlottingObserver should have access to this information, as to produce the right plot.
Running the whole callback list before the first epoch might not be desirable/possible; we might need to distinguish between MetricCallback and other callbacks.

guillaume-chevalier commented 3 years ago

@vincent-antaki Here is my suggestion:

Passing the Trainer to the AutoML constructor will allow to remove some arguments to the constructor of AutoML, as it has too many.
The new argument to know wheter or not to do this on the first loop is a trainer argument I believe. It seems it is the trainer's job to do this or not.
Doing so will also prepare for coding Hyperband later on, as coding Hyperband will require coding a new special Hyperband trainer (two birds w/ one stone).

Overall, allowing better injection and customization of the trainer to the AutoML loop will give the expected flexibility.

guillaume-chevalier commented 3 years ago

FYI @Eric2Hamel

guillaume-chevalier commented 3 years ago

vincent-antaki commented 3 years ago

@guillaume-chevalier I don't mind dependency inversion of Trainer/AutoML. However, your solution does not address point 2 and 3 of the consideration list. I'll implement it anyway because it is sufficient for what I need right now - and rather simple. But keep in mind point 2 and 3 are eventual problems to fix.

Eric2Hamel commented 3 years ago

What is the reason to calculate the metric before the first epoch?

vincent-antaki commented 3 years ago

Prettier training curves plot mainly :)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.