Closed Sllambias closed 8 months ago
@asbjrnmunk and @jakobamb if you've any QOL improvements you would like to see that are simple/medium difficult to implement add here
ad measure flops: When should this be run? It is a full forward pass, so it takes some time and would probably be a bit cumbersome to run always.
Otherwise sounds good.
@Sllambias for flop counting you can see https://github.com/asbjrnmunk/agave/blob/main/src/count_flops.py :)
reviews of course conditioned on passing automatic test and the manual tests i've written above
Todo:
[x] (WILL NOT IMPLEMENT) Add ThroughputMonitor REASON: No added value and probably moderate overhead. Measure FLOPs does what we want.
[x] Add measure_flops This is now printed when models are instantiated (also pr. layer and also including # params)
[x] Add option to overwrite in inference (so we can e.g. continue from broken runs) We can now use --overwrite if we wish to overwrite predictions AND results_json. Otherwise only non-existing will be predicted/evaluated
[x] Remove additional large keys in plans (new_sizes and new_spacings for now) - which means they will also not be in the ckpt and in wandb etc
[x] Add lm.hparams to Augmenter Now a part of the hparams.yaml and ckpt and wandb
[x] Recursive Find Function will now let you know if it doesn't find anything (e.g. if you've got a typo in the Planner or Manager specified in the CLI) Guess this is nice to find typos and bugs mainly
[x] Evaluator bug fix (correct results, but weird looking output for models with > 10 classes)
[x] WandB image plots Very beautiful. Need to figure out what to do exactly with classification. For now it is not enabled.
[x] (WILL NOT IMPLEMENT) Per label metrics during training REASON: known Torchmetrics bug, waiting for them to fix it https://github.com/Lightning-AI/torchmetrics/issues/2282
[x] Binary evaluation in inference Run yucca_evaluation with "--as_binary" to achieve this.
[x] Surface Dice (see: https://github.com/MIC-DKFZ/MedNeXt/blob/c5ed3f38b56d58c80581c75fea856865f42ddb75/nnunet_mednext/evaluation/region_based_evaluation.py#L112) Run yucca_evaluation with "--surface_eval" to achieve this
To be fully tested with (along with pytests):