New error on windows: RuntimeError: Encountered inf in predicted array

[wasserth]

Since yesterday I get the following error when running nnU-net v2.4.2 on windows. On ubuntu and mac this error does not happen. On windows this error also did not happen 3 days ago. Probably there was an update to some dependency without fixed version and this caused this new behaviour. Did you also come across this problem? Could it be a solution to clip inf values to the max number supported by the used dtype?

File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\totalsegmentator\nnunet.py", line 251, in nnUNetv2_predict
    predictor.predict_from_files(dir_in, dir_out,
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 256, in predict_from_files
    return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 373, in predict_from_data_iterator
    prediction = self.predict_logits_from_preprocessed_data(data).cpu()
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 490, in predict_logits_from_preprocessed_data
    prediction = self.predict_sliding_window_return_logits(data).to('cpu')
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 651, in predict_sliding_window_return_logits
    predicted_logits = self._internal_predict_sliding_window_return_logits(data, slicers,
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 607, in _internal_predict_sliding_window_return_logits
    raise e
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\nnunetv2\inference\predict_from_raw_data.py", line 600, in _internal_predict_sliding_window_return_logits
    raise RuntimeError('Encountered inf in predicted array. Aborting... If this problem persists, '
RuntimeError: Encountered inf in predicted array. Aborting... If this problem persists, reduce value_scaling_factor in compute_gaussian or increase the dtype of predicted_logits to fp32

[hari3100]

Same issue when running the totalsegmentator model on windows , i went down couple versions , v2.2.1 , it worked well.

[rw579]

Same issue when running the totalsegmentator model on windows , i went down couple versions , v2.2.1 , it worked well.

@hari3100 Can you share a pip list/freeze?

[hari3100]

@rw579 here it is :

Package Version

---

acvl_utils 0.2 asttokens 2.4.1 attrs 23.2.0 backports.tarfile 1.2.0 batchgenerators 0.25 batchgeneratorsv2 0.2 beautifulsoup4 4.12.3 bs4 0.0.2 certifi 2024.7.4 charset-normalizer 3.3.2 colorama 0.4.6 comm 0.2.2 connected-components-3d 3.18.0 contourpy 1.2.1 cycler 0.12.1 dataclasses 0.6 debugpy 1.8.2 decorator 5.1.1 dicom2nifti 2.4.11 dill 0.3.8 docutils 0.21.2 dunamai 1.18.0 dynamic_network_architectures 0.3.1 einops 0.8.0 exceptiongroup 1.2.2 executing 2.0.1 fft-conv-pytorch 1.2.0 filelock 3.15.4 fonttools 4.53.1 fsspec 2024.6.1 future 1.0.0 graphviz 0.20.3 idna 3.7 imagecodecs 2024.6.1 imageio 2.34.2 importlib_metadata 8.2.0 ipykernel 6.29.5 ipython 8.26.0 ipywidgets 8.1.3 jaraco.classes 3.4.0 jaraco.context 5.3.0 jaraco.functools 4.0.2 jedi 0.19.1 Jinja2 3.1.4 joblib 1.4.2 jsonschema 3.2.0 jupyter_client 8.6.2 jupyter_core 5.7.2 jupyterlab_widgets 3.0.11 keyring 25.3.0 kiwisolver 1.4.5 lazy_loader 0.4 linecache2 1.0.0 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.6.2 matplotlib-inline 0.1.7 mdurl 0.1.2 monai 1.3.2 more-itertools 10.3.0 mpmath 1.3.0 multiprocess 0.70.16 nest_asyncio 1.6.0 networkx 3.3 nh3 0.2.18 nibabel 5.0.0 nii2dcm 0.1.5 nnunetv2 2.5.1 numpy 1.23.2 opencv-python 4.10.0.84 p_tqdm 1.4.0 packaging 24.1 pandas 2.2.2 parso 0.8.4 pathos 0.3.2 pickleshare 0.7.5 pillow 10.4.0 pip 24.2 pkginfo 1.11.1 platformdirs 4.2.2 plotly 5.23.0 pox 0.3.4 ppft 1.7.6.8 prompt_toolkit 3.0.47 psutil 6.0.0 pure_eval 0.2.3 pyarrow 17.0.0 pydicom 2.3.0 pydicom-seg 0.4.1 Pygments 2.18.0 pyparsing 3.1.2 pyrsistent 0.20.0 python-dateutil 2.9.0 python-gdcm 3.0.24.1 pytz 2024.1 pywin32 306 pywin32-ctypes 0.2.2 PyYAML 6.0.1 pyzmq 26.0.3 readme_renderer 44.0 requests 2.32.3 requests-toolbelt 1.0.0 rfc3986 2.0.0 rich 13.7.1 rt-utils 1.2.7 scikit-image 0.24.0 scikit-learn 1.5.1 scipy 1.14.0 seaborn 0.13.2 setuptools 69.5.1 SimpleITK 2.3.1 six 1.16.0 soupsieve 2.5 stack-data 0.6.2 sympy 1.13.1 tcia_utils 2.1.15 tenacity 9.0.0 threadpoolctl 3.5.0 tifffile 2024.7.24 torch 2.4.0 tornado 6.4.1 TotalSegmentator 2.2.1 tqdm 4.66.4 traceback2 1.4.0 traitlets 5.14.3 twine 4.0.2 typing_extensions 4.12.2 tzdata 2024.1 Unidecode 1.3.8 unittest2 1.1.0 urllib3 2.2.2 wcwidth 0.2.13 wheel 0.43.0 widgetsnbextension 4.0.11 xvfbwrapper 0.2.9 yacs 0.1.8 zipp 3.19.2

[rw579]

@hari3100 thanks

[920773408]

@rw579 这里是：

运输版本

acvl_utils 0.2 asttokens 2.4.1 attrs 23.2.0 backports.tarfile 1.2.0 batchgenerators 0.25 batchgeneratorsv2 0.2 beautifulsoup4 4.12.3 bs4 0.0.2 certifi 2024.7.4 charset-normalizer 3.3.2 colorama 0.4.6 comm 0.2.2 Connected-Components-3d 3.18.0 contourpy 1.2.1 cycler 0.12.1 dataclasses 0.6 debugpy 1.8.2 decorator 5.1.1 dicom2nifti 2.4.11 dill 0.3.8 docutils 0.21.2 dunamai 1.18.0 dynamic_network_architectures 0.3.1 einops 0.8.0 exceptiongroup 1.2.2 执行 2.0.1 fft-conv-pytorch 1.2.0 filelock 3.15.4 fonttools 4.53.1 fsspec 2024.6.1 未来 1.0.0 graphviz 0.20.3 idna 3.7 imagecodecs 2024.6.1 imageio 2.34.2 importlib_metadata 8.2.0 ipykernel 6.29.5 ipython 8.26.0 ipywidgets 8.1.3 jaraco.classes 3.4.0 jaraco.context 5.3.0 jaraco.functools 4.0.2 jedi 0.19.1 Jinja2 3.1.4 joblib 1.4.2 jsonschema 3.2.0 jupyter_client 8.6.2 jupyter_core 5.7.2 jupyterlab_widgets 3.0.11 keyring 25.3.0 kiwisolver 1.4.5 lazy_loader 0.4 linecache2 1.0.0 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.6.2 matplotlib-inline 0.1.7 mdurl 0.1.2 monai 1.3.2 more-itertools 10.3.0 mpmath 1.3.0 multiprocess 0.70.16 nest_asyncio 1.6.0 networkx 3.3 nh3 0.2.18 nibabel 5.0.0 nii2dcm 0.1.5 nnunetv2 2.5.1 numpy 1.23.2 opencv-python 4.10.0.84 p_tqdm 1.4.0 包装 24.1 pandas 2.2.2 parso 0.8.4 pathos 0.3.2 pickleshare 0.7.5 pillow 10.4.0 pip 24.2 pkginfo 1.11.1 platformdirs 4.2.2 plotly 5.23.0 pox 0.3.4 ppft 1.7.6.8 prompt_toolkit 3.0.47 psutil 6.0.0 pure_eval 0.2.3 pyarrow 17.0.0 pydicom 2.3.0 pydicom-seg 0.4.1 Pygments 2.18.0 pyparsing 3.1.2 pyrsistent 0.20.0 python-dateutil 2.9.0 python-gdcm 3.0.24.1 pytz 2024.1 pywin32 306 pywin32-ctypes 0.2.2 PyYAML 6.0.1 pyzmq 26.0.3 readme_renderer 44.0 请求 2.32.3 请求工具带 1.0.0 rfc3986 2.0.0 rich 13.7.1 rt-utils 1.2.7 scikit-image 0.24.0 scikit-learn 1.5.1 scipy 1.14.0 seaborn 0.13.2 setuptools 69.5.1 SimpleITK 2.3.1 six 1.16.0 soupsieve 2.5 stack-data 0.6.2 sympy 1.13.1 tcia_utils 2.1.15 tenacity 9.0.0 threadpoolctl 3.5.0 tifffile 2024.7.24 torch 2.4.0 tornado 6.4.1 TotalSegmentator 2.2.1 tqdm 4.66.4 traceback2 1.4.0 traitlets 5.14.3 twine 4.0.2 typing_extensions 4.12.2 tzdata 2024.1 Unidecode 1.3.8 unittest2 1.1.0 urllib3 2.2.2 wcwidth 0.2.13 wheel 0.43.0 widgetsnbextension 4.0.11 xvfbwrapper 0.2.9 yacs 0.1.8 zipp 3.19.2

这个版本有点冲突把