Error building PyImath on Windows

[SimonBoorer]

I'm running into this error building PyImath on Windows: C:\src\Imath\src\python\PyImath\PyImathBasicTypes.cpp : fatal error C1128: number of sections exceeded object file format limit: compile with /bigobj

Adding /bigobj to _imath_extra_flags in config/LibraryDefine.cmake fixes it.

This might be specific to my build environment which slightly deviates from VFX Reference Platform CY2022:

• Windows SDK 10.0.18362
• MSVC 14.28.29333 (Visual Studio 2019)
• Python 3.10
• Boost Python 1.76

[cary-ilm]

@SimonBoorer, we could especially use the help with the Windows issues, our Windows expertise is limited.

[Andrej730]

met the same issue.

cmake .. -DPYTHON=ON -DBoost_USE_STATIC_LIBS=ON
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
-- Configure Imath 3.2.0-dev, library API version: 30.3.2.0
-- Imath pkg-config generation enabled
-- Found Python 3.11.3
-- Found Python3 libraries: 311
-- Will install to: Lib/site-packages
-- Disabling boost-provided cmake config. If this causes problems, consider setting Boost_NO_BOOST_CMAKE variable to OFF
-- Found Boost: C:/Boost/boost_1_83_0 (found version "1.83.0") found components: python311 missing components: python python3
-- Found Python 3 boost: Boost::python311
-- Found NumPy 1.23.4 for Python 3.11.3: Python311/Lib/site-packages/numpy/core/include
-- Boost PyImath
-- clang-format not found.
-- Configuring done (2.1s)
-- Generating done (0.4s)
-- Build files have been written to: /Imath/build

>msbuild .\INSTALL.vcxproj

"\Imath\build\INSTALL.vcxproj" (default target) (1) ->
"\Imath\build\ALL_BUILD.vcxproj" (default target) (3) ->
"\Imath\build\src\python\PyImath\imath_python3.vcxproj" (default target) (4) ->
"\Imath\build\src\python\PyImath\PyImath_Python3_11.vcxproj" (default target) (6) ->
(ClCompile target) ->
  \Imath\src\python\PyImath\PyImathBox.cpp : fatal  error C1128: number of sections exceeded obj
ect file format limit: compile with /bigobj [\Imath\build\src\python\PyImath\PyImath_Python3_11.
vcxproj]

    101 Warning(s)
    1 Error(s)

[Andrej730]

Found a workaround - run cmake with line cmake .. -DPYTHON=ON -DCMAKE_CXX_FLAGS="/DWIN32 /D_WINDOWS /W3 /GR /EHsc /bigobj" -DBoost_USE_STATIC_LIBS=ON where /bigobj is the compilator is asking for and /DWIN32 /D_WINDOWS /W3 /GR /EHsc is just default CMAKE_CXX_FLAGS values since we're overriding it.

It solved the /bigobj issue but it still failed to finish the compilation - it's looking for dynamic boost_python311 library even tough I've specified it to use static libs with -DBoost_USE_STATIC_LIBS=ON (static libs for boost are like libboost_python311).

"\Imath\build\ALL_BUILD.vcxproj" (default target) (3) ->
"\Imath\build\src\python\PyImath\imath_python3.vcxproj" (default target) (4) ->
"\Imath\build\src\python\PyImath\PyImath_Python3_11.vcxproj" (default target) (6) ->
(Link target) ->
  LINK : fatal error LNK1104: cannot open file 'boost_python311-vc143-mt-gd-x64-1_83.lib' [\Imat
h\build\src\python\PyImath\PyImath_Python3_11.vcxproj]

    179 Warning(s)
    1 Error(s)

[meshula]

Have you tried /bigobj but not Boost_USE_STATIC_LIBS? Boost_USE_STATIC_LIBS will cause trouble even if you get it linking because the boost python architecture expects that the type libraries are shared, and you'll find that if you try importing your resulting PyImath into something else that uses boost python, you will find things behaving very strangely; sometimes your code will work, and sometimes it won't, more or less depending on when and how the boost python bound types are resolved.

It's worrisome that you need /bigobj at all; PyImath hasn't got all that much in it, and we've run it on windows 20 years now without needing /bigobj. That said, I notice now that our CI does not build the python bindings on windows, that's not good; unless I'm looking in the wrong place?

[Andrej730]

Have you tried /bigobj but not Boost_USE_STATIC_LIBS?

Building it with dynamic libs and without /bigobj didn't worked. Turn out the problem was that I wasn't specifing release build with /p:configuration=release to msbuild and it was building debug build that was much bigger. I've tested both dynamic and static libs with /p:configuration=release and without /bigobj - both worked (though static libs still fails with cannot open file 'boost_python311-vc143-mt-gd-x64-1_83.lib').

Boost_USE_STATIC_LIBS will cause trouble even if you get it linking because the boost python architecture expects that the type libraries are shared, and you'll find that if you try importing your resulting PyImath into something else that uses boost python, you will find things behaving very strangely; sometimes your code will work, and sometimes it won't, more or less depending on when and how the boost python bound types are resolved.

You mean that if lib A imported to python is using let's say is using boost version A and lib B is using boost B, then it's possible that they both my end up using the same, let's say, boost A if lib A was imported first? I thought .pyd is connected to the specific .dll (e.g. mine imath.pyd is requiring boost_python311-vc143-mt-gd-x64-1_83.dll) so they shouldn't mess up.

Anyway, given the error above it seems that building it with static libs is not supported at the moment? Since it's end up requiring dynamic library.

That said, I notice now that our CI does not build the python bindings on windows, that's not good; unless I'm looking in the wrong place?

UPDATE! I've found the issue noticing python311_d.dll in the dumpbin outout. I've built Imath with msbuild .\INSTALL.vcxproj and it built debug build when I should have used msbuild .\INSTALL.vcxproj /p:configuration=release /p:platform=x64 - it worked fine and no errors on import. 🥳🥳

Already solved but might be useful for someone:

It worked but I've met another issue... It seems to be related to #238. Currently doing import imath in python 3.11 I get. Any ideas how to solve it?

Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

The code I'm using is below. I've also tried to copy everything to the same folder (both .pyd and .dll), same result.

import sys, os
sys.path.append(r"C:/Program Files (x86)/Imath/lib/site-packages")

# without os.add_dll_directory it throws:
# ImportError: DLL load failed while importing imath: The specified module could not be found.
os.add_dll_directory(r"C:/Boost/boost_1_83_0/stage/lib")
os.add_dll_directory(r"C:/Program Files (x86)/Imath/bin")

import imath # Fatal Python error: 

I've investigated imath.pyd with dumpbin. And all required dlls are accessible for python (before adding them with os.add_dll_directory it was raising ImportError: DLL load failed while importing imath.

Microsoft (R) COFF/PE Dumper Version 14.35.32217.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file C:\Program Files (x86)\Imath\lib\site-packages\imath.pyd

File Type: DLL

  Image has the following dependencies:

    python311_d.dll
    PyImath_Python3_11-3_2_d.dll
    Imath-3_2_d.dll
    boost_python311-vc143-mt-gd-x64-1_83.dll
    MSVCP140D.dll
    VCRUNTIME140D.dll
    VCRUNTIME140_1D.dll
    ucrtbased.dll
    KERNEL32.dll

[meshula]

Congratulations on your progress. I have not been brave enough to try python 3.11 yet.

The problem I was trying to describe is this ~ if the boost python libraries are statically linked, each pyd gets its own type dictionary in boost python.

What I encountered was that we tried to use PyImath with OpenTimelineIO and static linking, and we could write code along the lines of

V3f x = SomethingImathV3fFromOpenTimelineIO()
V3f y = SomethingImathV3fFromPyImath()
z = x + y

fails and Python complains that the types of x and y are unrelated. The reason is that an implementation detail in boost python is that it expects that the type tables are shared amongst pyds. It compares table objects for strict equality, instead of looking at the type data to discover that the types are the same on the basis of their name or whatever one might think it should do.

[Andrej730]

fails and Python complains that the types of x and y are unrelated. The reason is that an implementation detail in boost python is that it expects that the type tables are shared amongst pyds. It compares table objects for strict equality, instead of looking at the type data to discover that the types are the same on the basis of their name or whatever one might think it should do.

makes sense! so the problem is not that it will confuse one library for another but that it won't recognize that two libraries essentially use the same library and therefore they won't be able to interact with each other.

[meshula]

Yes, and interoperability was a goal we had :)