Bundles should handle zero validation input

[SachidanandAlle]

Is your feature request related to a problem? Please describe. For both single gpu and multi-gpu training, if the validation input is empty or not enough (in case of multi-gpu distribution) currently it throws an error. However it should allow and skip the Evaluation step if there is no enough data for validation. (example spleen bundle currently fails)

Describe the solution you'd like Many times, user has little data to finetune and existing model. And he/she doesn't have validate split or wants to skip.

Additional context Such situations normally happen during Active Learning iterations. Where annotators add small set of labels per iteration in the beginnning.

[Nic-Ma]

Thanks for raising the question. @yiheng-wang-nv Could you please help verify and double confirm what the problem is? At a first look, I think maybe we can disable validation by setting disabled for the ValidationHandler?

Thanks.

[yiheng-wang-nv]

Hi @Nic-Ma @SachidanandAlle ,

Take the spleen_ct_segmentation bundle for instance, skip validation just need to:

1. define a variable, like "skip_validation": true
2. add _disabled_ option in ValidationHandler, like "disabled": "@skip_validation"

However, I don't think it's a bundle issue here. Each bundle has its own training logic, some of them have validation set, some do not. The default settings are based on the original dataset and the purpose (of a bundle's author).

We can add a variable for a bundle that can let users skip the validation, but take spleen_ct_segmentation for instance, if skip the validation, how to determine when to save the checkpoint? CheckpointSaver is defined in "validate" and it will not be executed if skipping the validation. We can also add the this handler in "train", and trigger this handler only if skipping the validation. However, the next question is how to do model (checkpoint) selection? If handling all of the potential situations within a config file, it will include too much "extra" information.

Coming back to this ticket, my humble opinion is that to tell users how to skip the evaluation step, we can enhance this tutorial: https://github.com/Project-MONAI/tutorials/tree/main/model_zoo/adapt_bundle_to_another_dataset

Hi @SachidanandAlle @Nic-Ma @wyli @ericspod , what do you think? Thanks in advance.

[ericspod]

Hi @yiheng-wang-nv, yes each bundle does things differently so we'll be looking at best practices then rather than imposing requirements on bundles. If we're using validation to choose when to save we can set _disabled_ on a CheckpointSaver given to the trainer object like you say, it's adding a little redundant code but it's fine. In some situations we could use a subset of the training data as validation, this will be a problem when networks overfit on the training data and so report better validation metrics than is actually the general case but it would reduce the total number of saved states.