search

baal-org / baal

Bayesian active learning library for research and industrial usecases.
https://baal.readthedocs.io
Apache License 2.0
854 stars 84 forks source link

[Baal 2.0] All in one Experiment Object #288

Closed Dref360 closed 1 month ago

Dref360 commented 2 months ago

The active loop is pretty repetitive across experiments and is error prone as we know.

People forget to reset weights, log metrics, and check uncertainty. ActiveLearningLoop has extended its stay imo and we should deprecate it in favor of this new object. Baal would accommodate ModelWrapper and transformers.Trainer, but should make it easy to support new libraries as always.

So an experiment would now look like this:


wrapper = ModelWrapper(...)
wrapper.add_metric(...)

al_dataset = ActiveLearningDataset(...)
criterion = LabellingBudgetCriterion(...)

experiment = ActiveExperiment(trainer=wrapper, al_dataset=al_dataset, heuristic=EPIG(), query_size=100, criterion=criterion)

results = experiment.start()

And that's it! Optionally wrapper could be a transformers.Trainer.

To me this would give a bit of youth to our library that is now well over 5 years old :) We could deprecate ActiveLearningLoop in 2.0 and introduce Criterion, Experiment and EPIG for a massive release.