Open Keofire opened 2 weeks ago
SachidanandAlle
commented
2 weeks ago That's correct. The previous datalist is not updated when you retrain. It needs a change to do some bookkeeping of previously used train vs validation images and when you retrain again after adding more labels, it should maintain the previous pools for both validation and training.
Looking for a contribution to get it addressed - correctly. Please feel free to raise a PR if possible.
Keofire
commented
2 weeks ago I think the re-randomisation of the pools is not a problem from the learning point of view. I can imagine that it will give better results.
I see this phenomenon appearing strong, when a relatively short epoch number comes after a long one. My idea would be to skip the val_dice part for the x% of last runs epochs - this way, the scoring would be closer to reality.
I'm an orthopaedic surgeon. I'm quite far from implementing such a thing.
I'm using the slicer/monailabel duo with CT images (82) and going on the single label route (several times).
As I observed the training process, I realised a possible "cheat"/"error" in the system which can be misleading: While the segmentation process is going on, I use auto-learning; every time I upload a new label to the virtual server, it restarts the learning process. The validation dice is usually great in the first 3-10 epochs.
As the learning restarts, it resets the training/validation pool. I believe the "last times" training gets into the "next times" validation pool. This is not a big problem but probably gives misleading accuracy measures.