44 common interface for all metrics

alan-turing-institute / ARC-LoCoMoSeT

Low-Cost Model Selection for Transformers

MIT License

1 stars 0 forks source link

44 common interface for all metrics #59

Closed eddableheath closed 11 months ago

eddableheath commented 1 year ago

This pull request refactors the experiment pipeline to be centered around models, with the following features:

New config structure required.
Allows for single inference of model data shared across metrics that require same input.
Unifies metric structure.

New run.py script that allows for configs stating multiple models and random states.

jack89roberts commented 11 months ago

@eddableheath I've worked through this and changed a few things, can have a chat when you're back. Mostly it was tweaking docs/names, but I also refactored the metric classes a bit so most parameters are given to the constructor rather than to the metric fitting methods (that's more inline with sklearn, for example).

eddableheath commented 11 months ago

All looks good, runs and tests locally with the new changes too!