Multiple errors when building 1.2.4 (initially suspected to be ninja-related)

[TheChymera]

Due to Lua stability issues, I am building 1.2.5, which I understand is compatible with =lua-5.1.5.

I get the following full build.log, of which I assume this segment is the crux:

/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/Explicit/src/sitkExplicitITKImageToImageFilter.cxx:642:41: warning: type attributes ignored after type is already defined [-Wattributes]
  642 | template class SITKExplicit_EXPORT itk::ImageToImageFilter<itk::VectorImage<unsigned short, 3u>, itk::VectorImage<unsigned short, 3u> >;
      |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[621/1073] /usr/bin/x86_64-pc-linux-gnu-g++ -DITK_IO_FACTORY_REGISTER_MANAGER -DSITK_USE_EXPLICITITK -DSimpleITKIO_EXPORTS -ICode/IO/src/ITKFactoryRegistration -I/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/IO/include -I/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/IO/src -I/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/Common/include -ICode/Common/include -I/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/Explicit/include -isystem /usr/include/ITK-5.1  -march=native -O3 -pipe   -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -Wall -Wno-long-long -Wno-unused-local-typedefs -Wno-strict-overflow -Wextra -Wformat=2 -Wno-format-nonliteral -Wunused -Wpointer-arith -Winvalid-pch -Wcast-align -Wdisabled-optimization -Woverloaded-virtual -Wshadow -Wwrite-strings -Wstrict-null-sentinel -Wno-invalid-offsetof -MD -MT Code/IO/src/CMakeFiles/SimpleITKIO.dir/sitkImageFileReader.cxx.o -MF Code/IO/src/CMakeFiles/SimpleITKIO.dir/sitkImageFileReader.cxx.o.d -o Code/IO/src/CMakeFiles/SimpleITKIO.dir/sitkImageFileReader.cxx.o -c /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/IO/src/sitkImageFileReader.cxx
ninja: build stopped: subcommand failed.

Can you help me figure out what's wrong? I have =ninja-1.10.1 installed.

[blowekamp]

What version of ITK are you compiling against? ITK 1.2.4 was release build against ITK 4.13 with compatibility with ITK 5.0 when ITKV4_COMPATIBILITY is enabled.

The version of ITK being compiled against does not have the compatibility enabled, which causes this type of warning:

sitkImageReaderBase.cxx:282:28: error: could not convert ‘itk::ImageIOBase::USHORT’ from ‘const IOComponentEnum’ {aka ‘const itk::CommonEnums::IOComponent’} to ‘int’
  282 |     case itk::ImageIOBase::USHORT:
      |          ~~~~~~~~~~~~~~~~~~^~~~~~

/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Code/Explicit/src/sitkExplicitITKImageToImageFilter.cxx:642:41: warning: type attributes ignored after type is already defined [-Wattributes] 642 | template class SITKExplicit_EXPORT itk::ImageToImageFilter<itk::VectorImage<unsigned short, 3u>, itk::VectorImage<unsigned short, 3u> >; | ^~~~~~~~~~~~~~~~~~~~~~~

This compilation warning cat be addressed by configuring SimpleITK with SimpleITK_EXPLICIT_INSTANTIATION=OFF, (which is the default for SimpleITK 2.0 now).

With the above changes to configurations you may get a compilation error such as itkSLICImageFilter.h: No such file or directory, which can to be addressed by a change in code.

[TheChymera]

@blowekamp thank you for the clarification. Sadly none of the solutions seem to have worked:

(1) ITKv4 compatibility in ITK

This is the build.log after having built ITK with -DITKV4_COMPATIBILITY:BOOL=ON. Which looks pretty much the same... Any idea if I did something wrong? This is the cmake command used for ITK:

cmake -C /var/tmp/portage/sci-libs/itk-5.1.0/work/itk-5.1.0_build/gentoo_common_config.cmake
    -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_EXAMPLES=OFF -DBUILD_SHARED_LIBS=ON
    -DBUILD_TESTING=OFF -Ddouble-conversion_INCLUDE_DIRS=/usr/include/double-conversion
    -DExternalData_OBJECT_STORES=/var/tmp/portage/sci-libs/itk-5.1.0/work/InsightToolkit-5.1.0/.ExternalData
    -DGDCM_USE_SYSTEM_OPENJPEG=ON -DITK_FORBID_DOWNLOADS:BOOL=ON
    -DITK_INSTALL_LIBRARY_DIR=lib64 -DITK_USE_REVIEW=OFF -DITK_USE_SYSTEM_DCMTK=ON
    -DITK_USE_SYSTEM_DOUBLECONVERSION=ON -DITK_USE_SYSTEM_CASTXML=ON
    -DITK_USE_SYSTEM_HDF5=ON -DITK_USE_SYSTEM_JPEG=ON -DITK_USE_SYSTEM_PNG=ON
    -DITK_USE_SYSTEM_SWIG=ON -DITK_USE_SYSTEM_TIFF=ON -DITK_USE_SYSTEM_ZLIB=ON
    -DITK_USE_KWSTYLE=OFF -DITK_BUILD_DEFAULT_MODULES=ON -DITK_COMPUTER_MEMORY_SIZE=1
    -DITKV4_COMPATIBILITY:BOOL=ON -DModule_GenericLabelInterpolator:BOOL=ON
    -DModule_ITKReview:BOOL=ON -DWRAP_ITK_JAVA=OFF -DWRAP_ITK_TCL=OFF -DITK_WRAP_PYTHON=OFF
    -DCMAKE_BUILD_TYPE=Gentoo
    -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/sci-libs/itk-5.1.0/work/itk-5.1.0_build/gentoo_toolchain.cmake  /var/tmp/portage/sci-libs/itk-5.1.0/work/InsightToolkit-5.1.0

(2) SimpleITK Compilation with -DSimpleITK_EXPLICIT_INSTANTIATION=OFF

Gives me this build.log

(3) SimppleITK Compilation with -DSimpleITK_EXPLICIT_INSTANTIATION=OFF and patch

Using this patch:

From 9a9f67416683d69c1c8d2362ba6e50c4848803b2 Mon Sep 17 00:00:00 2001
From: Bradley Lowekamp <blowekamp@mail.nih.gov>
Date: Wed, 20 Nov 2019 15:30:50 -0500
Subject: [PATCH] Update SLICImageFilter's module

---
 Code/BasicFilters/json/SLICImageFilter.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Code/BasicFilters/json/SLICImageFilter.json b/Code/BasicFilters/json/SLICImageFilter.json
index 2ffba2f4c..0e9ced7d7 100644
--- a/Code/BasicFilters/json/SLICImageFilter.json
+++ b/Code/BasicFilters/json/SLICImageFilter.json
@@ -156,7 +156,7 @@
       ]
     }
   ],
-  "itk_module" : "SimpleITKFilters",
+  "itk_module" : "ITKSuperPixel",
   "detaileddescription" : "The Simple Linear Iterative Clustering (SLIC) algorithm groups pixels into a set of labeled regions or super-pixels. Super-pixels follow natural image boundaries, are compact, and are nearly uniform regions which can be used as a larger primitive for more efficient computation. The SLIC algorithm can be viewed as a spatially constrained iterative k-means method.\n\nThe original algorithm was designed to cluster on the joint domain of the images index space and it's CIELAB color space. This implementation works with images of arbitrary dimension as well as scalar, single channel, images and most multi-component image types including ITK's arbitrary length VectorImage .\n\nThe distance between a pixel and a cluster is the sum of squares of the difference between their joint range and domains ( index and value ). The computation is done in index space with scales provided by the SpatialProximityWeight parameters.\n\nThe output is a label image with each label representing a superpixel cluster. Every pixel in the output is labeled, and the starting label id is zero.\n\nThis code was contributed in the Insight Journal paper: \"Scalable Simple Linear Iterative Clustering (SSLIC) Using a\nGeneric and Parallel Approach\" by Lowekamp B. C., Chen D. T., Yaniv Z.",
   "briefdescription" : "Simple Linear Iterative Clustering (SLIC) super-pixel segmentation."
 }

Gives me this build.log.

Any ideas what else I can try?

[blowekamp]

Hello,

I believe I tested the above recomendation against ITK v5.0.1. I tested 5.1.0 and 5.1.1 and got similar issues as you report. Cherry picking commit: 73caeaa545859cffd68f9aebe1e8b66de5ab812d resolved the above issues.

Also add the following configuration option to ITK: Module_SimpleITKFilters:BOOL=ON

I would recommend spending effort on getting Lua 5.3 working, or using the SimpleITK superbuild/packaged version of Lua so that you can use a test and supported combination of SimpleITK/ITK.

[TheChymera]

I would recommend spending effort on getting Lua 5.3 working, or using the SimpleITK superbuild/packaged version of Lua so that you can use a test and supported combination of SimpleITK/ITK.

I am certainly looking to do that in the longer term :) , but for now that seems like it would put our usage of simpleITK on indefinite hold.

For the time being I have implemented all the suggestions from here, and I am getting this build.log. I have checked and the patches are applied correctly. The cmake command is:

cmake -C /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build/gentoo_common_config.cmake -G Ninja
   -DCMAKE_INSTALL_PREFIX=/usr -DUSE_SYSTEM_LUA=ON -DUSE_SYSTEM_SWIG=ON -DUSE_SYSTEM_GTEST=ON
   -DUSE_SYSTEM_ITK=ON -DUSE_SYSTEM_VIRTUALENV=ON -DSimpleITK_PYTHON_USE_VIRTUALENV:BOOL=OFF
   -DSimpleITK_EXPLICIT_INSTANTIATION=OFF -DModule_SimpleITKFilters:BOOL=ON
   -DCMAKE_BUILD_TYPE=Gentoo
   -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build
   /gentoo_toolchain.cmake  /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4

Also, just for reference, the respective cmake commands were at the top of all build logs I pasted thus far.

[blowekamp]

I see the following error:

CMake Error at /usr/share/cmake/Modules/ExternalData.cmake:1121 (message):

  Object MD5=f39ee5b792b0f3efb52d2601fe01022e not found at:

    https://s3.amazonaws.com/simpleitk/public/MD5/f39ee5b792b0f3efb52d2601fe01022e ("Couldn't resolve host name")
    https://simpleitk.github.io/SimpleITKExternalData/MD5/f39ee5b792b0f3efb52d2601fe01022e ("Couldn't resolve host name")
    https://data.kitware.com:443/api/v1/file/hashsum/MD5/f39ee5b792b0f3efb52d2601fe01022e/download ("Couldn't resolve host name")
    https://insightsoftwareconsortium.github.io/ITKTestingData/MD5/f39ee5b792b0f3efb52d2601fe01022e ("Couldn't resolve host name")
    https://itk.org/files/ExternalData/MD5/f39ee5b792b0f3efb52d2601fe01022e ("Couldn't resolve host name")

Please see this FAQ entry: https://simpleitk.readthedocs.io/en/master/faq.html#where-is-the-test-data

A tar-ball of the “SimpleITKData” can be downloaded for a release from the GitHub Assets, which contains the external data. It should populate the .ExternalData subdirectory of the SimpleITK source code directory when extracted.

Similar to ITK you can set "SimpleITK_FORBID_DOWNLOADS=ON" to disable the build from trying to download.

[TheChymera]

@blowekamp

Ok, trying to figure this out now, any idea however why I need the test data if I build simpleITK with

simpleitk-1.2.4_build/CMakeCache.txt
28:BUILD_TESTING:BOOL=OFF

?

[blowekamp]

Thank you for your continued work to get SimpleITK working on Gentoo.

$ git grep e26fd5c57ba45d6871ab2bd4e78ef8b7558036a5e30b40235f616e478d3997f3084e20c33033f132abe48ca7095239a195a9408a8bb6268d230f7cbd55e6d78b
Wrapping/R/Packaging/SimpleITK/man/sitkRdfiles.tar.gz.sha512:e26fd5c57ba45d6871ab2bd4e78ef8b7558036a5e30b40235f616e478d3997f3084e20c33033f132abe48ca7095239a195a9408a8bb6268d230f7cbd55e6d78b

The file being download is the R documentation archive. So the SimpleITKData archive is needed so that these required files can be found in the cache locally.

[TheChymera]

@blowekamp Thanks for the explanation.

Saldy the build still fails with:

cd /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build/Wrapping/R && /usr/bin/cmake -Drelative_top=/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build -Dfile=/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build/ExternalData/Wrapping/R/Packaging/SimpleITK/man/sitkRdfiles.tar.gz -Dname=/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/SimpleITK-1.2.4/Wrapping/R/Packaging/SimpleITK/man/sitkRdfiles.tar.gz -Dexts=.md5+.sha512 -DExternalData_ACTION=fetch -DExternalData_CONFIG=/var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build/Wrapping/R/SimpleITKRpackageData_config.cmake -P /usr/share/cmake/Modules/ExternalData.cmake
CMake Error at /usr/share/cmake/Modules/ExternalData.cmake:1121 (message):

  Object MD5=f39ee5b792b0f3efb52d2601fe01022e not found at:

    (No ExternalData_URL_TEMPLATES given)

  Object
  SHA512=e26fd5c57ba45d6871ab2bd4e78ef8b7558036a5e30b40235f616e478d3997f3084e20c33033f132abe48ca7095239a195a9408a8bb6268d230f7cbd55e6d78b
  not found at:

    (No ExternalData_URL_TEMPLATES given)

Full build.log.

I have checked, and the data is copied in the correct location in the build directory (tree with more levels is just too much text :) ):

silenthost /var/tmp/portage/sci-libs/simpleitk-1.2.4/work # tree -L 1 simpleitk-1.2.4_build/.ExternalData/
simpleitk-1.2.4_build/.ExternalData/
├── MD5
├── README.rst
└── SHA512

[TheChymera]

Trying to debug a bit more, it seems like I have both the checksum-name file and the sitkRdfiles*-name file. Any idea why it wouldn't just work then?

silenthost /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build # find -name "*f39ee5b792b0f3efb52d2601fe01022e*"
./.ExternalData/MD5/f39ee5b792b0f3efb52d2601fe01022e
silenthost /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build # find ../ -name "*sitkRdfiles*"
../SimpleITK-1.2.4/Wrapping/R/Packaging/SimpleITK/man/sitkRdfiles.tar.gz.md5
../SimpleITK-1.2.4/Wrapping/R/Packaging/SimpleITK/man/sitkRdfiles.tar.gz.sha512

[TheChymera]

@blowekamp Should I perhaps set any of these values explicitly?

silenthost /var/tmp/portage/sci-libs/simpleitk-1.2.4/work/simpleitk-1.2.4_build # ag ExternalData_URL_TEMPLATES
CMakeCache.txt
338:ExternalData_URL_TEMPLATES:STRING=
993://ADVANCED property for variable: ExternalData_URL_TEMPLATES
994:ExternalData_URL_TEMPLATES-ADVANCED:INTERNAL=1

Wrapping/R/SimpleITKRpackageData_config.cmake
2:set(ExternalData_URL_TEMPLATES "")

[blowekamp]

From here: https://github.com/SimpleITK/SimpleITK/blob/1720d80024aa1527f372163ac5dff4e44068c80f/CMake/sitkExternalData.cmake#L5 You may want to set "ExternalData_OBJECT_STORES". I have not had a chance (🦃 ) to replicate your build and determine a solution, but inspecting and setting the path to the objects stores is where I would start.

[TheChymera]

@blowekamp ok, this option for cmake seems to have gotten it to compile:

-DExternalData_OBJECT_STORES:STRING="${BUILD_DIR}/.ExternalData"

I do however now have some issues with the resulting files. This might be an issue regarding compliance with Portage (the Gentoo package manager), and outside your purview, but if you do have any ideas, I'd be very happy to hear them. As it stands I have no clue what the issue might be: build.log.

This being the relevant bit:

Files matching a file type that is not allowed:
   usr/lib/libSimpleITK_ITKLevelSets-1.2.so.1
   usr/lib/libSimpleITKBasicFilters0-1.2.so.1
   usr/lib/libSimpleITK_ITKWatersheds-1.2.so.1
   usr/lib/libSimpleITK_ITKAnisotropicSmoothing-1.2.so.1
   usr/lib/libSimpleITK_ITKImageFeature-1.2.so.1
   usr/lib/libSimpleITK_ITKImageFusion-1.2.so.1
   usr/lib/libSimpleITK_ITKImageGrid-1.2.so.1
   usr/lib/libSimpleITK_ITKDeconvolution-1.2.so.1
   usr/lib/libSimpleITK_ITKSmoothing-1.2.so.1
   usr/lib/libSimpleITK_ITKSuperPixel-1.2.so.1
   usr/lib/libSimpleITK_ITKMathematicalMorphology-1.2.so.1
   usr/lib/libSimpleITK_ITKBiasCorrection-1.2.so.1
   usr/lib/libSimpleITK_ITKLabelVoting-1.2.so.1
   usr/lib/libSimpleITK_ITKReview-1.2.so.1
   usr/lib/libSimpleITK_ITKImageSources-1.2.so.1
   usr/lib/libSimpleITK_ITKDisplacementField-1.2.so.1
   usr/lib/libSimpleITK_ITKFastMarching-1.2.so.1
   usr/lib/libSimpleITK_ITKCommon-1.2.so.1
   usr/lib/libSimpleITK_ITKImageCompare-1.2.so.1
   usr/lib/libSimpleITKCommon-1.2.so.1
   usr/lib/libSimpleITK_ITKThresholding-1.2.so.1
   usr/lib/libSimpleITK_ITKTransform-1.2.so.1
   usr/lib/libSimpleITK_ITKImageNoise-1.2.so.1
   usr/lib/libSimpleITK_ITKBinaryMathematicalMorphology-1.2.so.1
   usr/lib/libSimpleITK_ITKImageFunction-1.2.so.1
   usr/lib/libSimpleITKBasicFilters1-1.2.so.1
   usr/lib/libSimpleITK_ITKImageGradient-1.2.so.1
   usr/lib/libSimpleITKIO-1.2.so.1
   usr/lib/libSimpleITK_ITKAntiAlias-1.2.so.1
   usr/lib/libSimpleITK_ITKPDEDeformableRegistration-1.2.so.1
   usr/lib/libSimpleITK_ITKCurvatureFlow-1.2.so.1
   usr/lib/libSimpleITK_ITKImageIntensity-1.2.so.1
   usr/lib/libSimpleITK_ITKDenoising-1.2.so.1
   usr/lib/libSimpleITK_ITKImageFilterBase-1.2.so.1
   usr/lib/libSimpleITK_ITKLabelMap-1.2.so.1
   usr/lib/libSimpleITK_ITKRegistrationCommon-1.2.so.1
   usr/lib/libSimpleITK_ITKImageLabel-1.2.so.1
   usr/lib/libSimpleITK_ITKRegionGrowing-1.2.so.1
   usr/lib/libSimpleITK_ITKImageStatistics-1.2.so.1
   usr/lib/libSimpleITKRegistration-1.2.so.1
   usr/lib/libSimpleITK_ITKFFT-1.2.so.1
   usr/lib/libSimpleITK_ITKImageCompose-1.2.so.1
   usr/lib/libSimpleITK_ITKDistanceMap-1.2.so.1
   usr/lib/libSimpleITK_ITKClassifiers-1.2.so.1
   usr/lib/libSimpleITK_ITKConvolution-1.2.so.1
   usr/lib/libSimpleITK_ITKConnectedComponents-1.2.so.1
   usr/lib/libSimpleITK_ITKColormap-1.2.so.1

[TheChymera]

@blowekamp

Ok, it seems I needed to set the lib directory so that 64bit libs don't end up in /usr/lib but rather /usr/lib64:

-DSimpleITK_INSTALL_LIBRARY_DIR=$(get_libdir)

get_libdir being a Portage function.

Looks good now. Thank you so much for assisting with this! I might get back with issues regarding package behaviour once installed, though :)

[blowekamp]

Outstanding new you got the build working. 🎆

A build feature that may be needed, if you need to wrap for multiple version of a language is briefly described here: https://simpleitk.readthedocs.io/en/master/building.html#advanced-build-options

Basically, after the core SimpleITK libraries and header are built and installed, the language sub-directories under the "Wrapping" directory can be build as separate projects dependent on the SimpleITK core.