search

Project-MONAI / tutorials

MONAI Tutorials
https://monai.io/started.html
Apache License 2.0
1.76k stars 666 forks source link

Too large `step_size` in deepedit #1430

Open KumoLiu opened 1 year ago

KumoLiu commented 1 year ago

Describe the bug Too large step_size which may cause the learning rate never decreases unless the model is trained for 5000 epochs. https://github.com/Project-MONAI/tutorials/blob/bbc4e180f5130859396a98e35523bc73fa694595/deepedit/ignite/train.py#L268

Expected behavior Set smaller step_size or set epoch_level=False in LrScheduleHandler

Originally posted in https://github.com/Project-MONAI/MONAI/issues/6625

KumoLiu commented 1 year ago

Hi @diazandr3s, could you please help take a look at this issue? Thanks in advance!

diazandr3s commented 1 year ago

Hi @KumoLiu,

Thanks for the ping. Good point! Have you tried reducing the step_size? you may also want to consider changing the learning rate and/or optimizer. As you may know, intensity normalization may also affect the training.

KumoLiu commented 1 year ago

Hi @diazandr3s, I didn't try to reduce the step_size, I'm assuming there's a typo here because the value of 5000 is set too high if it's epoch-level. And if it's a typo, I can submit a PR to fix it.

Thanks!