search

benchopt / benchmark_resnet_classif

Benchopt benchmark for ResNet fitting on a classification task
https://benchopt.github.io/results/benchmark_resnet_classif.html
8 stars 4 forks source link

ENH: add validation split #22

Closed pierreablin closed 2 years ago

pierreablin commented 2 years ago

This PR adds a validation split to the train / val / test. The test data should only be used to report final perf, validation set is to be used to early stop.

pierreablin commented 2 years ago

This is a WIP

pierreablin commented 2 years ago

@tomMoral yes this is what I'll try to do next.

tomMoral commented 2 years ago

An idea:

why not compute a matching first between the datasets in torch and tf, store it a file (potentially on OSF but I am fine in the repo to start with), and fix this order when loading the data?

that way we can do the train/valid split in the objective. This also solves the issue with the slow data consistency check as the order should always be the same, so no need to recompute it anymore.

WDYT @zaccharieramzi ?

zaccharieramzi commented 2 years ago

I think it's a good idea

zaccharieramzi commented 2 years ago

All green in local

zaccharieramzi commented 2 years ago

all green in local, just this problem of the CIFAR DL again, submitting a tentative PR to fix the robustness

zaccharieramzi commented 2 years ago

no time to wait for the macos workers