Running build_usd.py with --openimageio fails because it can't find boost

[LucaScheller]

Description of Issue

Hey, I'm currently trying to build Usd via the build_script.py and it fails when building with openimagio.

Running this: python USD/build_scripts/build_usd.py USD/build works fine (except also having this issue in the UsdView: https://github.com/PixarAnimationStudios/USD/issues/2153)

Running this: python USD/build_scripts/build_usd.py USD/build --openimageio fails with the following error:

Building with settings:
  USD source directory          /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD
  USD install directory         /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build
  3rd-party source directory    /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/src
  3rd-party install directory   /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build
  Build directory               /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/build
  CMake generator               Default
  CMake toolset                 Default
  Downloader                    curl

  Building                      Shared libraries
    Variant                     Release
    Target                      
    Imaging                     On
      Ptex support:             Off
      OpenVDB support:          Off
      OpenImageIO support:      On 
      OpenColorIO support:      Off 
      PRMan support:            Off
    UsdImaging                  On
      usdview:                  On
    Python support              On
      Python Debug:             Off
      Python 3:                 On
    Documentation               Off
    Tests                       Off
    Examples                    On
    Tutorials                   On
    Tools                       On
    Alembic Plugin              Off
      HDF5 support:             Off
    Draco Plugin                Off
    MaterialX Plugin            Off

  Dependencies                  OpenImageIO
STATUS: Installing OpenImageIO...
2023-01-12 23:30
cmake -DCMAKE_INSTALL_PREFIX="/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build" -DCMAKE_PREFIX_PATH="/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build" -DCMAKE_BUILD_TYPE=Release    -DOIIO_BUILD_TOOLS=OFF -DOIIO_BUILD_TESTS=OFF -DUSE_PYTHON=OFF -DSTOP_ON_WARNING=OFF -DOPENEXR_ROOT="/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build" -DUSE_PTEX=OFF -DBoost_NO_BOOST_CMAKE=On -DBoost_NO_SYSTEM_PATHS=True "/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/src/oiio-Release-2.1.16.0"
-- The CXX compiler identification is GNU 12.2.1
-- The C compiler identification is GNU 12.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Building OpenImageIO 2.1.16.0
-- CMake version is 3.25.1
-- Configuring OpenImageIO 2.1.16.0
-- CMake 3.25.1
-- CMake system           = Linux-6.1.3-200.rog.fc37.x86_64
-- CMake system name      = Linux
-- Project source dir     = /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/src/oiio-Release-2.1.16.0
-- Project build dir      = /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/build/oiio-Release-2.1.16.0
-- Project install prefix = /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build
-- Configuration types    = 
-- Build type             = Release
-- Setting Namespace to: OpenImageIO_v2_1
-- CMAKE_CXX_COMPILER     = /usr/bin/c++
-- CMAKE_CXX_COMPILER_ID  = GNU
-- Building for C++11
-- clang-format found: /usr/bin/clang-format
-- 
-- * Checking for dependencies...
-- *   - Missing a dependency 'Package'?
-- *     Try cmake -DPackage_ROOT=path or set environment var Package_ROOT=path
-- *   - To exclude an optional dependency (even if found),
-- *     -DUSE_Package=OFF or set environment var USE_Package=OFF 
-- 
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: thread) (found suitable version "1.70.0",
  minimum required is "1.53")
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindBoost.cmake:2376 (find_package_handle_standard_args)
  src/cmake/externalpackages.cmake:90 (find_package)
  src/cmake/externalpackages.cmake:164 (checked_find_package)
  CMakeLists.txt:117 (include)

It seems to not find the boost install that the previous build step created. I've tried adding it via

extraArgs.append('-DBoost_ROOT="/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/build/src/boost_1_70_0"')

on line 1332 in build_script.py, but with no success.

System Information (OS, Hardware)

OS: Fedora 37 CPU: AMD Ryzen™ 9 6900HX Python Version: Python 3.9.16

Has anyone had this issue so far? Any help would be highly appreciated :) Cheers, Luca

[sunyab]

Hi @LucaScheller, I'm guessing that you had built once into a directory without OIIO enabled, then tried to build again into the same directory with OIIO enabled -- does that sound right? If so, you're running into a deficiency in the build script where it doesn't realize boost has to be rebuilt to include the boost::threads library that OIIO needs.

Could you try building with --openimageio into a fresh directory, or adding --force BOOST to your command line if you're building into the same directory?

[LucaScheller]

Hey, sunyab thanks for the fast reply. If I build it on the first run with openimageio I get this error. log.txt

I'll try to build it again with the exact versions mentioed here: https://github.com/PixarAnimationStudios/USD/blob/release/VERSIONS.md

My current cmake and gcc versions are a lot newer, that might cause the problems

[sunyab]

Ah yeah, from the log it looks like boost is hitting build errors when compiling boost::thread. I can't find any reports of this but I wonder if boost 1.70 has problems with gcc 12. You could try changing the version of boost getting used in build_usd.py to something later and see if that helps?

[lgritz]

OIIO CI tests against gcc12, so that should be fine from the OIIO perspective. Also, OIIO can use any boost from 1.53 or so up through the very latest (its use of boost is extremely minimal, and I am aiming to eventually wean it off the last few things, but we're not quite there yet).

Looks from the logs like your boost build itself is failing, maybe something in the old 1.70 boost is not making gcc12 happy. The oiio build log at the top of this thread indicates that oiio is finding boost 1.70, but that in particular it does not appear to contain the thread module as expected. (If you're using a pre-built boost from a package installer, sometimes boost can be divided up and you may need to request boost-threads or boost-all explicitly?)

My take is that vfx platform may say boost 1.70 or whatever, but actually very little changes from one boost version to the next (other than adding new libraries) and most of their modules are header only, so there is usually no real harm in having a newer boost than they expect. If you need a newer boost because you simply must use gcc 12, that's probably fine (note: vfx platform says it tries to be compatible with gcc 9 for 2022, gcc 11 for 2023, which nobody is using yet, so insisting on gcc12 may come back to haunt you as you build other packages that are trying to fit into VFX Platform, it might all work, or you might find yourself trying to swim upstream).

It also may be that the errors gcc is complaining about when building boost are actually just escalated warnings that can be individually disabled. I'm no expert in building boost, but maybe their build scripts have some kind of "please don't treat warnings as full errors" option that can be used to just force it to build despite the problems?

Aside: Does USD's CI test against gcc11 and gcc12? I recommend it. Even if nobody is using them in the studios today, it's usually better to get all those new compiler warnings cleaned up long before people are relying on it.

[LucaScheller]

Hey @lgritz / @sunyab, thanks for your replies!

I'm using the build script (build_scripts/build_usd.py) shipped with the repo that downloads and builds with boost 1.70.0. It builds boost without errors when not building with oiio. According to the buildscript the threading modules are being built.

If I change the boost version to 1.81.0 (current latest), it builds oiio successfully, but fails to build Usd (I get the attached error log). With version 1.74.0 it builds successfully :tada: (Not sure if it is worth doing a PR for this, as I'm not sticking to the VFX Ref Platform)

When using the aswf/ci-base:2020.9 docker image, it also builds without problems. If I run into more problems though (when e.g. building with OCIO etc), I think I'll just compile via docker then, to avoid the headaches :)

Thanks again for your help so far :)

| Does USD's CI test against gcc11 and gcc12? Not sure if this is being done, sounds like a thing worth doing though

log.txt

[meshula]

The patch suggested by the compiler in the log.txt might address the 1.81.0 problem and would be harmless otherwise, if you're motivated to give it a shot.

In file included from /mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/pxr/base/gf/wrapPlane.cpp:26:
/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/pxr/base/gf/plane.h:207:36: error: ‘vector’ in namespace ‘std’ does not name a template type
  207 | bool GfFitPlaneToPoints(const std::vector<GfVec3d>& points, GfPlane* fitPlane);
      |                                    ^~~~~~
/mnt/data/SOFTWARE/USD___Latest___x64___linux/USD/pxr/base/gf/plane.h:33:1: note: ‘std::vector’ is defined in header ‘<vector>’; did you forget to ‘#include <vector>’?
   32 | #include "pxr/base/gf/api.h"
  +++ |+#include <vector>
   33 | 

[sunyab]

Filed as internal issue #USD-7870