search

RTKConsortium / RTK

Reconstruction Toolkit
Apache License 2.0
246 stars 145 forks source link

Python Wrapping fails for missing dimensions #575

Closed MrTzschr closed 10 months ago

MrTzschr commented 10 months ago

Hi,

I am building RTK with the following settings:

cmake -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=OFF -DModule_CudaCommon:BOOL=ON -DModule_CudaCommon_GIT_TAG:STRING=v1.0.1 -DITK_WRAP_PYTHON:BOOL=ON -DModule_RTK:BOOL=ON -DRTK_USE_CUDA:BOOL=ON -DRTK_BUILD_APPLICATIONS:BOOL=OFF -GNinja ../src

This results in the following error during compilation:

Generating rtkZengForwardProjectionImageFilterPython.cpp, ../../Generators/Python/itk/rtkZengForwardProjectionImageFilterPython.py
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:305: Warning 401: Nothing known about base class 'itk::ImageSource< itk::CudaImage< itk::Vector< float,2 >,4 > >'. Ignored.
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:305: Warning 401: Maybe you forgot to instantiate 'itk::ImageSource< itk::CudaImage< itk::Vector< float,2 >,4 > >' using %template.
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:455: Warning 401: Nothing known about base class 'itk::ImageSource< itk::CudaImage< itk::Vector< float,3 >,4 > >'. Ignored.
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:455: Warning 401: Maybe you forgot to instantiate 'itk::ImageSource< itk::CudaImage< itk::Vector< float,3 >,4 > >' using %template.
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:605: Warning 401: Nothing known about base class 'itk::ImageSource< itk::CudaImage< itk::Vector< float,4 >,4 > >'. Ignored.
C:/ITK/test/Wrapping/Typedefs/itkImageToImageFilterRTK.i:605: Warning 401: Maybe you forgot to instantiate 'itk::ImageSource< itk::CudaImage< itk::Vector< float,4 >,4 > >' using %template.
make[2]: *** [Wrapping/Modules/RTK/CMakeFiles/RTKPython.dir/build.make:1583: Wrapping/Modules/RTK/rtkZengForwardProjectionImageFilterPython.cpp] Error 6
make[1]: *** [CMakeFiles/Makefile2:34438: Wrapping/Modules/RTK/CMakeFiles/RTKPython.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

This is just one example and happens for a lot of classes. I tried a similar build in WSL running Ubuntu, but with python 3.10.12, and the same errors appear, so it seems platform independent.

A possible solution seems to request the (missing?) 4th dimension by setting -DITK_WRAP_IMAGE_DIMS:String="2;3;4" but I assume the idea of https://github.com/RTKConsortium/RTK/blob/65f0b66f4ec6fb9a5754977ddb9c8df99267b3e9/wrapping/itkImageSourceRTK.wrap#L44-L46 was to generate the missing parts automagically.

My environment is: Windows 10 VS 2019 v16.11.32 Python v3.8.18 Ninja v1.11.1

SimonRit commented 10 months ago

Thanks for the report. I am already struggling to maintain the Python wrapping functional with the default external module options. I am using

cmake $SCRIPT_PATH/itk \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DBUILD_EXAMPLES=OFF \
    -DBUILD_SHARED_LIBS=OFF \
    -DBUILD_TESTING=OFF \
    -DITK_WRAP_PYTHON=ON \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_PATH \
    -DModule_RTK:BOOL=ON \
    -DModule_RTK_GIT_TAG="" \
    -DModule_CudaCommon:BOOL=ON \
    -DModule_CudaCommon_GIT_TAG="" \
    -DRTK_USE_CUDA:BOOL=ON \
    -DRTK_BUILD_APPLICATIONS:BOOL=OFF \
    -DPY_SITE_PACKAGES_PATH:STRING=${BINDINGS_PATH} \
    -DITK_WRAP_unsigned_short:BOOL=ON \
    -DITK_WRAP_double:BOOL=ON \
    -DITK_WRAP_complex_double:BOOL=ON \
    -DITK_WRAP_IMAGE_DIMS:STRING="2;3;4"

to mimick ITK (see here I think). That will be hard to manage all possible combinations of CMake options but the default options should be better managed, I'll try to fix it.

SimonRit commented 10 months ago

Can you check that #576 fixes the issue for your too?

MrTzschr commented 10 months ago

Works. Thanks for the fast fix!

MrTzschr commented 10 months ago

At first I tested this with seperate builds of ITK and RTK to apply the patch. Now, after the merge, I compiled RTK as module of ITK by setting the commit tag explicitly and end up with a lot of warnings on executing python code. The code itself works without issues but the warning appears on every code execution :/ Compilation went without errors.

Template itk::CudaImage<itk::Vector<float,5>,4>
 already defined as <class 'itk.itkCudaImageRTKPython.itkCudaImageVF54'>
 is redefined as <class 'itk.itkCudaImageRTKPython.itkCudaImageVF54'>
Warning: template already defined 'itk::CudaImage<itk::Vector<float,5>,4>'
Template itk::CudaImage<itk::CovariantVector<float,5>,4>
 already defined as <class 'itk.itkCudaImageRTKPython.itkCudaImageCVF54'>
 is redefined as <class 'itk.itkCudaImageRTKPython.itkCudaImageCVF54'>
Warning: template already defined 'itk::CudaImage<itk::CovariantVector<float,5>,4>'
Template itk::ImageSource<itk::CudaImage<float,2>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIF2'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIF2'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<float,2>>'
Template itk::ImageSource<itk::CudaImage<float,3>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIF3'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIF3'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<float,3>>'
Template itk::ImageSource<itk::CudaImage<float,4>>
 already defined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIF4'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIF4'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<float,4>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,2>,2>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF22'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF22'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,2>,2>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,2>,3>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF23'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF23'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,2>,3>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,3>,2>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF32'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF32'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,3>,2>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,3>,3>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF33'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF33'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,3>,3>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,4>,2>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF42'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF42'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,4>,2>>'
Template itk::ImageSource<itk::CudaImage<itk::Vector<float,4>,3>>
 already defined as <class 'itk.itkImageSourceCudaCommonPython.itkImageSourceCIVF43'>
 is redefined as <class 'itk.itkImageSourceRTKPython.itkImageSourceCIVF43'>
Warning: template already defined 'itk::ImageSource<itk::CudaImage<itk::Vector<float,4>,3>>'
Template itk::CudaImageDataManager<itk::CudaImage<itk::Vector<float,5>,4>>
 already defined as <class 'itk.itkCudaImageDataManagerRTKPython.itkCudaImageDataManagerCIVF54'>
 is redefined as <class 'itk.itkCudaImageDataManagerRTKPython.itkCudaImageDataManagerCIVF54'>
Warning: template already defined 'itk::CudaImageDataManager<itk::CudaImage<itk::Vector<float,5>,4>>'
Template itk::CudaImageDataManager<itk::CudaImage<itk::CovariantVector<float,5>,4>>
 already defined as <class 'itk.itkCudaImageDataManagerRTKPython.itkCudaImageDataManagerCICVF54'>
 is redefined as <class 'itk.itkCudaImageDataManagerRTKPython.itkCudaImageDataManagerCICVF54'>
Warning: template already defined 'itk::CudaImageDataManager<itk::CudaImage<itk::CovariantVector<float,5>,4>>'

EDIT: I repeated the ninja test and must have done something different in the first test. The same wall of warnings appears here. Sorry for the confusion. Maybe one of your compiler flags removes the warnings?

SimonRit commented 10 months ago

Sorry about that... Should be addressed by #577 which requires RTKConsortium/ITKCudaCommon#31, i.e. -DModule_CudaCommon_GIT_TAG:STRING=9d3fe9bdedbac76abc38ea61beff4158de65109f".

SimonRit commented 10 months ago

CI-compiled packages (CudaCommon + RTK) work fine, I'm closing the issue but reopen if I missed something (again).

MrTzschr commented 10 months ago

I am afraid it is still not working as intended. I noticed f5aab1f68e53061264f4f6f3af5e0d49c5f7481d bumps the ITK modules to the required versions but I still end up with the warnings.

Perhaps my description was too imprecise. The warnings appear when running python code, not during compilation, e.g. when calling:

import itk
from itk import RTK as rtk
rtk.Image
SimonRit commented 10 months ago

So you are using ITK master and you have removed -DModule_CudaCommon_GIT_TAG:STRING=v1.0.1 from your cmake command?

MrTzschr commented 10 months ago

Yes, for the most recent test. For completeness I used this:

cmake -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=OFF -DITK_WRAP_PYTHON:BOOL=ON -DModule_RTK:BOOL=ON -DRTK_USE_CUDA:BOOL=ON -DRTK_BUILD_APPLICATIONS:BOOL=OFF -GNinja ../src
SimonRit commented 10 months ago

Ok, I don't know what happened, I probably did not test the right version because I can reproduce the issue... Re-opening. Thanks for your patience.

SimonRit commented 10 months ago

Hopefully #578 will do

MrTzschr commented 10 months ago

Is there an easy way to test this fix as part of the RTK module in ITK? I tried replacing RTK.remote.cmake with

  GIT_REPOSITORY https://github.com/SimonRit/RTK.git
  GIT_TAG DefaultWrapping

but ended up with

-- NOTE: Using override 'Module_RTK_GIT_TAG=e9caad5b4409ded681bc1f39d76a1bdc7985f782'
      instead of value 'GIT_TAG=DefaultWrapping'
      specified in file C:/ITK/v5.4rc02/src/Modules/Remote/RTK.remote.cmake'
From https://github.com/SimonRit/RTK
 * branch              HEAD       -> FETCH_HEAD
fatal: reference is not a tree: e9caad5b4409ded681bc1f39d76a1bdc7985f782
CMake Error at CMake/ITKModuleRemote.cmake:111 (message):
  Failed to checkout tag: 'e9caad5b4409ded681bc1f39d76a1bdc7985f782'
Call Stack (most recent call first):
  CMake/ITKModuleRemote.cmake:145 (_git_update)
  CMake/ITKModuleRemote.cmake:274 (_fetch_with_git)
  Modules/Remote/RTK.remote.cmake:40 (itk_fetch_module)
  Modules/Remote/CMakeLists.txt:21 (include)

during configuring.

The second attempt was to build ITK and RTK separately by:

cmake -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=OFF -DModule_CudaCommon:BOOL=ON -DITK_WRAP_PYTHON:BOOL=ON -GNinja ../src-test

and

cmake -DRTK_USE_CUDA:BOOL=ON -DRTK_BUILD_APPLICATIONS:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=OF -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=OFF -DITK_DIR:STRING="C://ITK/bin-test" -GNinja ../rtk-src

but during runtime from python I encountered:

ImportError: DLL load failed while importing _RTKPython

I tried fixing this by adding os.add_dll_directory('C:/ITK/rtk-bin/bin') to the FirstCudaReconstruction example but then ended with

RuntimeError: C:\ITK\src-test\Modules\Filtering\FFT\include\itkRealToHalfHermitianForwardFFTImageFilter.h:85:
ITK ERROR: Object factory failed to instantiate class itk::RealToHalfHermitianForwardFFTImageFilter<class itk::Image<float,3>,class itk::Image<class std::complex<float>,3> >

Is there something I missed for the seperate builds?

SimonRit commented 10 months ago

If you change RTK.remote.cmake (which is a good idea), you should remove -DModule_RTK_GIT_TAG from the command line (which you haven't since there is the message Using override 'Module_RTK_GIT_TAG=e9caad5b4409ded681bc1f39d76a1bdc7985f782').

I don't get the python load error but I don't compile it separately usually.

MrTzschr commented 10 months ago

Oh boy, cached CMake variable. With a fresh build it works. Thanks a lot for the help and fixes!

SimonRit commented 10 months ago

Fixed by #578 578