search

InsightSoftwareConsortium / ITK

Insight Toolkit (ITK) -- Official Repository. ITK builds on a proven, spatially-oriented architecture for processing, segmentation, and registration of scientific images in two, three, or more dimensions.
https://itk.org
Apache License 2.0
1.4k stars 664 forks source link

Python wrappings do not compile with Ninja on Windows 11 #3216

Closed dzenanz closed 2 years ago

dzenanz commented 2 years ago

Python wrapping fail to compile with Ninja generator when using MSVC compile from VS2019 on Windows 11.

Steps to Reproduce

  1. Start x64 developer command prompt of VS2019
  2. Make a new directory next to ITK source
  3. C:\Dev\ITK-ninja>cmake -DITK_WRAP_PYTHON=ON -GNinja ..\ITK-git
  4. C:\Dev\ITK-ninja>ninja

Expected behavior

Compiles without errors.

Actual behavior

...
-- [download 100% complete]
-- verifying file...
       file='C:/Dev/ITK-ninja/Wrapping/Generators/SwigInterface/swig-prefix/src/archive.tar'
-- Downloading... done
-- extracting...
     src='C:/Dev/ITK-ninja/Wrapping/Generators/SwigInterface/swig-prefix/src/archive.tar'
     dst='C:/Dev/ITK-ninja/Wrapping/Generators/SwigInterface/swigwin-4.0.2'
-- extracting... [tar xf]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[5439/8182] Generating ../../itkArray.xml
FAILED: Wrapping/itkArray.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkArray.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkArray.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5441/8182] Generating ../../itkContinuousIndex.xml
FAILED: Wrapping/itkContinuousIndex.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkContinuousIndex.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkContinuousIndex.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5444/8182] Generating ../../itkDefaultDynamicMeshTraits.xml
FAILED: Wrapping/itkDefaultDynamicMeshTraits.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkDefaultDynamicMeshTraits.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkDefaultDynamicMeshTraits.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5445/8182] Generating ../../itkDiffusionTensor3D.xml
FAILED: Wrapping/itkDiffusionTensor3D.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkDiffusionTensor3D.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkDiffusionTensor3D.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5448/8182] Generating ../../itkExtractImageFilter.xml
FAILED: Wrapping/itkExtractImageFilter.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkExtractImageFilter.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkExtractImageFilter.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5450/8182] Generating ../../itkFunctionBase.xml
FAILED: Wrapping/itkFunctionBase.xml
cmd.exe /C "cd /D C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon && C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\bin\castxml.exe -o C:/Dev/ITK-ninja/Wrapping/itkFunctionBase.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-msvc ( "C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe" /DWIN32 /D_WINDOWS /GR /EHsc -W3 /bigobj /bigobj -std:c++14 ) -fexceptions -w -c @C:/Dev/ITK-ninja/Wrapping/ITKCommon.castxml.inc C:/Dev/ITK-ninja/Wrapping/itkFunctionBase.cxx"
error: '--castxml-cc-msvc' compiler command failed:

 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '/DWIN32' '/D_WINDOWS' '/GR' '/EHsc' '-W3' '/bigobj' '/bigobj' '-std:c++14' '-c' '-FoNUL' 'C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml/share/castxml/detect_vs.cpp'
detect_vs.cpp

#define _CPPRTTI 1
#define _CPPUNWIND 1
#define _INTEGRAL_MAX_BITS 64
#define _MSC_BUILD 0
#define _MSC_EXTENSIONS 1
#define _MSC_FULL_VER 192930140
#define _MSC_VER 1929
#define _MSVC_LANG 201402L
#define _MT 1
#define _M_AMD64 100
#define _M_X64 100
#define _NATIVE_WCHAR_T_DEFINED 1
#define _WCHAR_T_DEFINED 1
#define _WIN32 1
#define _WIN64 1
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 16ull
#define __STDCPP_THREADS__ 1
#define __cplusplus 199711L
class type_info;
C:\Dev\ITK-ninja\Wrapping\Generators\CastXML\castxml\share\castxml\detect_vs.cpp : fatal error C1083: Cannot open compiler generated file: 'C:\Dev\ITK-ninja\Wrapping\Modules\ITKCommon\NUL.obj': Permission denied

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

[5456/8182] Generating pyBasePython.cpp, ../itk/pyBasePython.py
ninja: build stopped: subcommand failed.

C:\Dev\ITK-ninja>

Reproducibility

Versions

Environment

Multiple versions of ITK and CMake. Key ingredients are ninja and Windows 11.

bradking commented 2 years ago

castxml internally runs the compiler command given to --castxml-cc-msvc in order to detect its preprocessing behavior. In this case, it's running cl ... -FoNUL ..., where the -FoNUL option is supposed to suppress generating any actual output file. Windows has a special hook to name the NUL device (equivalent to /dev/null) in as a file in any directory. Maybe something changed in Windows 11.

Try creating an empty foo.c source file, and run cl -c foo.c -FoNUL at the command prompt manually.

aylward commented 2 years ago

The result of cl -c foo.c -FoNUL is an object file called NUL.obj

C:\src\castxml_test>cl -c foo.c -FoNUL
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30140 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

foo.c

C:\src\castxml_test>dir
 Volume in drive C is OS
 Volume Serial Number is 1A9E-6A0B

 Directory of C:\src\castxml_test

03/07/2022  01:33 PM    <DIR>          .
03/07/2022  01:30 PM    <DIR>          ..
03/07/2022  01:33 PM                 0 foo.c
03/07/2022  01:33 PM               485 NUL.obj
               3 File(s)            490 bytes
               2 Dir(s)  742,621,663,232 bytes free
dzenanz commented 2 years ago

I just checked: echo blah >null.txt produces a file on Windows 11, but not on Windows 10. echo blah >null does not produce a file on Windows 11 either.

bradking commented 2 years ago

@aylward @dzenanz please visit HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Null in regedit and check the value of the Start setting.

dzenanz commented 2 years ago

On both Windows 10 and 11, in HKLM\SYSTEM\ControlSet001\Services\Null, Start is 0x00000001.

bradking commented 2 years ago

@dzenanz thanks. Please try your echo blah >nul.txt experiment with the name nul.txt rather than null.txt. The NUL device name is supposed to work with any extension, IIUC. We've been relying on -FoNUL causing cl to write to NUL.obj, which should also be the null device.

dzenanz commented 2 years ago

Sorry, I meant nul.txt, not null.txt. So:

echo blah >nul.txt produces a file on Windows 11, but not on Windows 10. echo blah >nul does not produce a file on Windows 11 either.

I accidently copied null instead of nul earlier, because I was trying both.

bradking commented 2 years ago

Thanks. We can't prevent cl from adding the .obj suffix, so I'll look into another approach.

bradking commented 2 years ago

I opened CastXML/CastXML#211 for this.

bradking commented 2 years ago

I released CastXML 0.4.5 with an update to avoid depending on -FoNUL.

dzenanz commented 2 years ago

We should update CastXML in ITK.

dzenanz commented 2 years ago

@aylward did you run into this? You could try with an external build of CastXML, by turning ITK_USE_SYSTEM_CASTXML on.

thewtex commented 2 years ago

Binaries for CastXML 0.4.5 are being built.

thewtex commented 2 years ago

Binaries updated in #3324