Open michalwols opened 4 years ago
Hey @michalwols.
Yes, metadata related to training is something we were thinking of as a future feature. We have some ideas here, but would be interested to hear your thoughts about a lightweight way to capture this might be? (presently we are focussed on post-training metadata that can be extracted from pretrained models).
Just adding a params key to the evaluator that takes a dict should work.
If you want to keep it clean you could use a TypedDict (it's available in mypy_extensions) to define the supported set of keys and their types.
class ImageClassificationParams(TypedDict):
label_smoothing: bool
optimizer: str
warmup: bool
epochs: int
batch_size: int
weight_decay: float
extra_data: bool
from sotabencheval.image_classification import ImageClassificationParams, ImageNetEvaluator
evaluator = ImageNetEvaluator(
model_name='FixResNeXt-101 32x48d',
paper_arxiv_id='1906.06423',
params=ImageClassificationParams( # could also just be a dict if you annotate the function with params: ImageClassificationParams
epochs=200,
batch_size=32,
weight_decay=.1,
extra_data=True,
fixres=True # new key introduced by this method
)
)
I guess for enum/choice support something like a dataclass might be better than a typed dict.
This looks like an awesome project.
Would be great if there was a way to report hyper parameters with each submission.