search

microsoft / InnerEye-DeepLearning

Medical Imaging Deep Learning library to train and deploy 3D segmentation models on Azure Machine Learning
https://aka.ms/innereyeoss
MIT License
558 stars 142 forks source link

Benchmark the evaluation of a segmentation model on a full test set image #479

Open ant0nsc opened 3 years ago

ant0nsc commented 3 years ago

It appears to take very long to run the ensemble model on the test set, even though we are using crops that are as large as the full image. Inference on 50 prostate images takes 6 hours total (Run 3304). Doing the forward propagation of the scan through all 5 models takes most time, almost all of the 6 hours. Computing the metrics off all stored results takes only minutes.

AB#4096

ant0nsc commented 3 years ago

Potential culprit: Before running a full image inference on the GPU, we split a model across all GPUs. Is it possible that we are doing that too often? The model split should be computed only once, not repeatedly for each test subject.