microsoft / InnerEye-DeepLearning

Medical Imaging Deep Learning library to train and deploy 3D segmentation models on Azure Machine Learning
https://aka.ms/innereyeoss
MIT License
553 stars 142 forks source link

Error during inference if patch size is larger than image #676

Open fepegar opened 2 years ago

fepegar commented 2 years ago

If the patch size is larger than the image size during inference with TorchIO, an error is raised. This implies that this situation wasn't being covered by the tests.

To do:

Error:

Traceback (most recent call last):
  File "InnerEyePrivate/ML/runner.py", line 50, in <module>
    main()
  File "InnerEyePrivate/ML/runner.py", line 46, in main
    post_cross_validation_hook=runner.default_post_cross_validation_hook)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/runner.py", line 451, in run
    return runner.run()
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/runner.py", line 218, in run
    self.run_in_situ(azure_run_info)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/runner.py", line 407, in run_in_situ
    self.ml_runner.run()
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/run_ml.py", line 351, in run
    self.run_inference(checkpoint_paths_for_testing, ModelProcessing.DEFAULT)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/run_ml.py", line 474, in run_inference
    model_proc=model_proc)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/run_ml.py", line 698, in model_inference_train_and_test
    model_proc=model_proc)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/model_testing.py", line 72, in model_test
    return segmentation_model_test(config, data_split, checkpoint_paths, model_proc)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/model_testing.py", line 102, in segmentation_model_test
    epoch_and_split=epoch_and_split)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/model_testing.py", line 166, in segmentation_model_test_epoch
    voxel_spacing_mm=sample.metadata.image_header.spacing
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/pipelines/inference.py", line 47, in predict_and_post_process_whole_image
    return self.post_process(self.predict_whole_image(image_channels, voxel_spacing_mm, mask, patient_id))
  File "/azureml-envs/azureml_019d9b5c5814e5f53d425814bccd13a8/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/azureml/cr/j/5a59e32ccd764940b8143a7327f35cb5/exe/wd/innereye-deeplearning/InnerEye/ML/pipelines/inference.py", line 272, in predict_whole_image
    padding_mode=self.model_config.padding_mode.value,
  File "/azureml-envs/azureml_019d9b5c5814e5f53d425814bccd13a8/lib/python3.7/site-packages/torchio/data/sampler/grid.py", line 77, in __init__
    self.locations = self._compute_locations(self.subject)
  File "/azureml-envs/azureml_019d9b5c5814e5f53d425814bccd13a8/lib/python3.7/site-packages/torchio/data/sampler/grid.py", line 102, in _compute_locations
    self._parse_sizes(*sizes)
  File "/azureml-envs/azureml_019d9b5c5814e5f53d425814bccd13a8/lib/python3.7/site-packages/torchio/data/sampler/grid.py", line 131, in _parse_sizes
    raise ValueError(message)
ValueError: Patch size (48, 64, 48) cannot be larger than image size (35, 51, 35)

AB#5114

fepegar commented 2 years ago

For some reason, this was not automatically corrected by

https://github.com/microsoft/InnerEye-DeepLearning/blob/821cb3be7a1b9cebbd06d49481e2024aa7cd0b06/InnerEye/ML/models/architectures/base_model.py#L77-L80