search

InsightSoftwareConsortium / ITK

Insight Toolkit (ITK) -- Official Repository. ITK builds on a proven, spatially-oriented architecture for processing, segmentation, and registration of scientific images in two, three, or more dimensions.
https://itk.org
Apache License 2.0
1.43k stars 668 forks source link

ITK Python v5.3rc03 fails to import ITKImageGrid after upgrade #3175

Open tbirdso opened 2 years ago

tbirdso commented 2 years ago

Description

Seeing an issue where ITK v5.3rc03 fails to find the ImageGrid module after upgrading from a previous ITK version such as v5.2.post1. Succeeds on a clean install.

Steps to Reproduce

  1. pip install itk==v5.3rc03
  2. Launch Python console
  3. import itk
  4. itk.auto_progress(2) (for module visibility)
  5. itk.force_load()

Expected behavior

All modules load without error

Actual behavior

>>> itk.force_load()
Loading ITKPyBase... done
Loading ITKCommon... done
Loading ITKStatistics... done
Loading ITKImageFilterBase... done
Loading ITKTransform... done
Loading ITKMesh... done
Loading ITKImageFunction... done
Loading ITKSpatialObjects... done
Loading ITKSmoothing... done
Loading ITKOptimizers... done
Loading ITKPath... done
Loading ITKImageCompose... done
Loading ITKImageStatistics... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/init_helpers.py", line 116, in force_load
    getattr(itk, k)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 98, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageGrid'
>>> itk.ImageGrid
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/lazy.py", line 130, in __getattribute__
    value = types.ModuleType.__getattribute__(self, attr)
AttributeError: module 'itk' has no attribute 'ImageGrid'

Reproducibility

After upgrading from v5.2.1.post1: 100% After a clean install (uninstall and install all ITK packages with pip): 0%

Versions

ITK 5.3 release candidate 3

Environment

Observed on Windows and Linux in Python 3.8

Additional Information

Observed after direct upgrade with pip install itk==v5.3rc03 on both Windows and Linux. Problem was resolved for both platforms by removing all ITK packages with pip uninstall itk ... and reinstalling with pip install itk.

tbirdso commented 2 years ago

@dzenanz If you have time could you please verify whether you can reproduce? I haven't seen an upgrade path issue for ITK like this before. Simple enough to re-install, but I imagine this may be something we want to resolve before the 5.3 release to avoid forcing existing users to un- and re-install ITK Python modules.

tbirdso commented 2 years ago

@thewtex Ping for visibility

dzenanz commented 2 years ago

I can reproduce it. Windows 10, Python 3.9.6. Going from 5.2.1.post1 to 5.3rc3 gets me:

...
Loading ITKImageStatistics... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\site-packages\itk\support\init_helpers.py", line 116, in force_load
    getattr(itk, k)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 98, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageGrid'

After pip uninstall itk and cleaning up all ITK-related folders in site-packages, then pip install itk==v5.3rc03 gets a properly working ITK.

tbirdso commented 2 years ago

I don't see any commit messages in ITKImageGrid since 5.3rc02 was tagged on Oct 27th that should changed Python module behavior 🤔

ITKImageGrid history

tag history

dzenanz commented 2 years ago

It could be due to some changes in ITKCommon or some Python-related infrastructure.

tbirdso commented 2 years ago

Observed a different error loading ITKFFT this morning that was resolved again by re-installing ITK packages. Haven't tried recreating with an upgrade path.

>>> import itk
>>> itk.__version__
'5.3.0'
>>> itk.auto_progress(2)
>>> itk.Forward1DFFTImageFilter
Loading ITKPyBase... done
Loading ITKCommon... done
Loading ITKStatistics... done
Loading ITKImageFilterBase... done
Loading ITKTransform... done
Loading ITKImageFunction... done
Loading ITKImageGrid... done
Loading ITKFFT... Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 110, in itk_load_swig_module
    l_module = loader.load(swig_module_name)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 259, in load
    l_spec.loader.exec_module(l_module)  # pytype: disable=attribute-error
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\..\ITKFFTPython.py", line 85, in <module>
    from itk.itkForward1DFFTImageFilterPython import *
ModuleNotFoundError: No module named 'itk.itkForward1DFFTImageFilterPython'
thewtex commented 2 years ago

It may be that pip is not resolving package dependencies correctly.

It is best to start from a clean virtual environment / conda environment.

PranjalSahu commented 2 years ago

Faced the same issue with ITK ImageGrid. Works fine in new conda environment.

tbirdso commented 2 years ago

Currently seeing a similar issue again with ITKImageSources in my local environment for v5.3rc04:

>>> import itk
>>> itk.auto_progress(2)
>>> itk.DisplacementFieldTransform
Loading ITKPyBase... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\lazy.py", line 138, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 96, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 96, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 92, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageSources'

Note that investigation in base.py and site-packages/itk reveal that itk_base_global_module_data is populated via the <module>Config.py files present in itk/Configuration. It appears that ITKImageSourcesConfig.py is not present in my Configuration folder and is not installed along with other config files when I re-install itk-filtering. ITKImageSourcesConfig.py is present in another virtual environment's site-packages/itk/Configuration folder for v5.3rc04, and v5.3rc04 has not failed for me before now, so this does not seem to be as straightforward as the file not being included in the package. Perhaps something being cached strangely?

EDIT: Note that simply copying over this config file does not resolve the issue. No files related to ITKImageSources appear to be present in my site-packages/itk folder.

EDIT2: As before, pip uninstalling all ITK packages and re-installing from cache apparently resolves the problem locally.

tbirdso commented 2 years ago

Note: recently observed by @PranjalSahu via pip install itk in Slicer.

jcfr commented 2 years ago

After comparing the site-packages folder obtained in the following two scenario:

.. we can see that filters added by 5.3rc3 are not installed when upgrading:

image

jcfr commented 2 years ago

Complete list of missing files after performing the upgrade from 5.2.1.post1 to 5.3rc3:

> itk/Configuration/ITKFFTConfig.py
> itk/Configuration/ITKFFT_snake_case.py
> itk/Configuration/ITKImageGridConfig.py
> itk/Configuration/ITKImageGrid_snake_case.py
> itk/itkBinShrinkImageFilterPython.py
> itk/itkBSplineDownsampleImageFilterPython.py
> itk/itkBSplineUpsampleImageFilterPython.py
> itk/itkChangeInformationImageFilterPython.py
> itk/itkComplexToComplexFFTImageFilterPython.py
> itk/itkConstantPadImageFilterPython.py
> itk/itkCropImageFilterPython.py
> itk/itkCyclicShiftImageFilterPython.py
> itk/itkExpandImageFilterPython.py
> itk/itkFFTPadImageFilterPython.py
> itk/ITKFFTPython.py
> itk/_ITKFFTPython.so
> itk/itkFFTShiftImageFilterPython.py
> itk/itkFFTWComplexToComplexFFTImageFilterPython.py
> itk/itkFFTWForwardFFTImageFilterPython.py
> itk/itkFFTWGlobalConfigurationPython.py
> itk/itkFFTWHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkFFTWInverseFFTImageFilterPython.py
> itk/itkFFTWRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkFlipImageFilterPython.py
> itk/itkForwardFFTImageFilterPython.py
> itk/itkFullToHalfHermitianImageFilterPython.py
> itk/itkHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkHalfToFullHermitianImageFilterPython.py
> itk/ITKImageGridPython.py
> itk/_ITKImageGridPython.so
> itk/itkInterpolateImageFilterPython.py
> itk/itkInverseFFTImageFilterPython.py
> itk/itkMirrorPadImageFilterPython.py
> itk/itkOrientImageFilterPython.py
> itk/itkPadImageFilterBasePython.py
> itk/itkPadImageFilterPython.py
> itk/itkPasteImageFilterPython.py
> itk/itkPermuteAxesImageFilterPython.py
> itk/itkRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkRegionOfInterestImageFilterPython.py
> itk/itkResampleImageFilterPython.py
> itk/itkShrinkImageFilterPython.py
> itk/itkSliceBySliceImageFilterPython.py
> itk/itkTileImageFilterPython.py
> itk/itkVnlComplexToComplexFFTImageFilterPython.py
> itk/itkVnlForwardFFTImageFilterPython.py
> itk/itkVnlHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkVnlInverseFFTImageFilterPython.py
> itk/itkVnlRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkWarpImageFilterPython.py
> itk/itkWrapPadImageFilterPython.py
> itk/itkZeroFluxNeumannPadImageFilterPython.py

List was gathered on Linux in a python3.8 environment

For reference, here is how the list of files copied above was generated:

cd ..
cd site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3
rm -rf itk/__pycache__ itk/support/__pycache__ itk/Configuration/__pycache__
fd . | sort > /tmp/site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3.txt

cd site-packages-with-itk-5.3rc3
fd __pycache__
rm -rf itk/__pycache__ itk/support/__pycache__ itk/Configuration/__pycache__
fd . | sort  > /tmp/site-packages-with-itk-5.3rc3.txt

diff \
  /tmp/site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3.txt \
  /tmp/site-packages-with-itk-5.3rc3.txt | ack "[>|<]"