Open ant0nsc opened 3 years ago
Potential culprit: Before running a full image inference on the GPU, we split a model across all GPUs. Is it possible that we are doing that too often? The model split should be computed only once, not repeatedly for each test subject.
It appears to take very long to run the ensemble model on the test set, even though we are using crops that are as large as the full image. Inference on 50 prostate images takes 6 hours total (Run 3304). Doing the forward propagation of the scan through all 5 models takes most time, almost all of the 6 hours. Computing the metrics off all stored results takes only minutes.
AB#4096