search

Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.37k stars 3.38k forks source link

Cross validation feature #839

Closed BraveDistribution closed 2 years ago

BraveDistribution commented 4 years ago

🚀 Feature

Cross-Validation is a crucial model validation techniques for assessing how the model generalizes on new data.

Motivation

Research papers usually require cross-validation. From my point of view, this kind of feature would simplify the work of researches.

Pitch

I want to pass a parameter to the Trainer object to specify that I want to train the model on K-folds.

In the case that nobody wants to make a PR, I can start working on that.

function2-llx commented 2 years ago

@chanshing The example code on the master branch is inconsistent with tag 1.5.10, if you're using 1.5.10 you should refer to code with this tag.

pchalasani commented 2 years ago

after the first fold, the subsequent ones don't run at all.. do you have any idea why that might be?

I noticed this too, with early stopping. The subsequent fold training loops retain state from the first fold, and so the behavior is as if the early stopping condition is already satisfied, and hence they don't run.

Even for the MNIST example given, due to the max_epochs=10 param, after the fold trains, the subsequent ones start at epoch=9 and don't really do anything. You can see this by printing the value of self.fit_loop.current_epoch just prior to self.fit_loop.run() in the advance() function. It prints 9 for the subsequent folds. Note that there is no early stopping used here.

(All these refer to the version pointed out by @function2-llx above ).

There needs to be a way to truly reset the fit_loop state as each fold training starts (so that callbacks like Early Stopping, ModelCheckpoint, and stopping conditions like max_epochs function propertly). I tried doing self.fit_loop.reset() in the _reset_fitting() function but that did not help.

YerePhy commented 2 years ago

Hi, I would like what is the KFold feature state in Lightning since I am really interested but the example provided in the official documentation is no longer available

tchaton commented 2 years ago

Hey @YerePhy

To my knowledge, the best version of KFold is this repo: https://github.com/SkafteNicki/pl_cross.

Best, T.C

YerePhy commented 2 years ago

Thanks @tchaton, looks great, I will give a try for sure.

YerePhy commented 2 years ago

Hey, pl_cross is not maintained and there are some issues with >=1.6... the example of KFold is no longer available in the official documentation of Loops API... will this feature be implemented anytime soon? It would be very very helpful for the medical imaging community!

peterchristofferholm commented 1 year ago

Some of the issues that @YerePhy is refering to are described here

peterchristofferholm commented 1 year ago

@rohitgr7 could this issue be reopened?

YaduKini commented 1 year ago

Hello, Is there a document about how to implement K-Fold cross validation in Lightning? Any pointers would be much appreciated. Thanks in advance.

jannik-el commented 6 months ago

Any updates on this feature? I can't currently find any information on if there is a lightning 2.0 officially supported way to do this, would be nice to know before I get stuck into a hacky way of doing it for my project.