search

autonlab / auton-survival

Auton Survival - an open source package for Regression, Counterfactual Estimation, Evaluation and Phenotyping with Censored Time-to-Events
http://autonlab.github.io/auton-survival
MIT License
316 stars 74 forks source link

Early stopping? #47

Closed Jesse-Islam closed 3 years ago

Jesse-Islam commented 3 years ago

Hi,

I was wondering if there's an early stopping parameter for dsm that's already implemented, or an ideal work around that you would use now to get it going. I've been digging through the source code and can't seem to find any. I imagine its not too complicated as you're extending pytorch, but I'm also not sure what the best way to make it extend your functions would be.

chiragnagpal commented 3 years ago

@Jesse-Islam The code does implement a somewhat memory inefficient version of early stopping, check lines 187-207 https://github.com/autonlab/DeepSurvivalMachines/blob/2801c6c63a48195821733245c85f9a90bd21d446/dsm/utilities.py#L187https://github.com/autonlab/DeepSurvivalMachines/blob/2801c6c63a48195821733245c85f9a90bd21d446/dsm/utilities.py#L187 @Jeanselme is working on a more efficient version here: #37

Jesse-Islam commented 3 years ago

Awesome! So I could add a minimum delta loss required and increase patience from 2 to whatever i'd like...

Thanks for the help!

chiragnagpal commented 3 years ago

sure, anytime :) you can also pass a tuple of validation data to val_data to compute the loss on a heldout validation set

Jesse-Islam commented 3 years ago

Would you suggest that over using the vsize parameter? for example setting it to .20 so that the validation would be 20% of the data fed in to train.

To clarify this is more a question on the implementation differences for dsm rather than the machine learning best practice

chiragnagpal commented 3 years ago

either is ok -- visze would randomly sample the val set from the training data.. if you want something that is reproducible across runs, i'd recommend just pass the val_data explicitly