search

adobe / USD-Fileformat-plugins

A collection of USD fileformat plugins
286 stars 20 forks source link

Build failed due to cmake find_python3 module #29

Open issacclee opened 2 weeks ago

issacclee commented 2 weeks ago

Hi,

I'm building USD-Fileformat-plugins in docker. I used the prebuilt usd artifact from github releases. And I tried to mimicking the way github ci of this repo is setup. However, when I tried to build the docker image, it fails at the find python module part of the cmake file. Here is the dockerfile:

FROM python:3.10.14-slim-bookworm
WORKDIR /app

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    ca-certificates \
    git \
    unzip \
    curl \
    cmake \
    build-essential 

#download the prebuilt usd and place it in workdir
ADD http://demo.uality.cn/usd-2405-ubuntu-22.04.zip /app/usd.zip

RUN unzip /app/usd.zip -d /app/usd

ADD https://github.com/adobe/USD-Fileformat-plugins/archive/refs/tags/v1.0.5.zip /app/release.zip

RUN unzip /app/release.zip -d /app/

WORKDIR /app/USD-Fileformat-plugins-1.0.5

RUN apt-get install -y python3-dev

#ADD http://demo.uality.cn/pxrConfig.cmake /app/usd/pxrConfig.cmake

#ADD http://demo.uality.cn/pxrTargets.cmake /app/usd/cmake/pxrTargets.cmake

ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/usd/bin:/app/usd/lib:/app/usd/lib64:/app/usd/plugin/usd:/app/USD-Fileformat-plugins-1.0.5/bin/bin:/app/USD-Fileformat-plugins-1.0.5/bin/lib:/app/USD-Fileformat-plugins-1.0.5/bin/plugin/usd:/user/local/lib
ENV USD_BUILD_PATH=/app/usd
ENV PYTHONPATH=$PYTHONPATH:/app/usd/lib/python
ENV PXR_PLUGINPATH_NAME =/app/USD-Fileformat-plugins-1.0.5/bin/plugin/usd:/app/usd/plugin/usd
ENV pythonLocation=/usr/local/bin/python
ENV PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
ENV Python_ROOT_DIR=/usr/local

RUN cmake -S . -B build -DCMAKE_INSTALL_PREFIX=bin -DCMAKE_BUILD_TYPE=Release -Dpxr_ROOT=/app/usd -DUSD_FILEFORMATS_BUILD_TESTS=OFF -DUSD_FILEFORMATS_ENABLE_FBX=OFF -DUSD_FILEFORMATS_ENABLE_GLTF=ON -DUSD_FILEFORMATS_ENABLE_SBSAR=OFF -DUSD_FILEFORMATS_ENABLE_OBJ=OFF -DUSD_FILEFORMATS_ENABLE_PLY=OFF -DUSD_FILEFORMATS_ENABLE_STL=OFF -DUSD_FILEFORMATS_FETCH_GTEST=OFF -DUSD_FILEFORMATS_FETCH_HAPPLY=OFF -DUSD_FILEFORMATS_FETCH_FMT=OFF -DUSD_FILEFORMATS_FETCH_FASTFLOAT=OFF -DUSD_FILEFORMATS_ENABLE_CXX11_ABI=ON -DOpenImageIO_INCLUDE_DIR=/app/usd/include -DOpenImageIO_INCLUDES=/app/usd/include -DOpenImageIO_LIB_DIR=/app/usd/lib -DPython3_INCLUDE_DIR=/usr/local/include/python3.10 -DPython3_LIBRARY=/usr/local/lib

RUN cmake --build build
RUN cmake --install build

ENTRYPOINT [ "/app/usd/bin/usdcat" ]

And this is the terminal output I get when I tried to build the image

0.323 -- The C compiler identification is GNU 12.2.0
0.384 -- The CXX compiler identification is GNU 12.2.0
0.397 -- Detecting C compiler ABI info
0.473 -- Detecting C compiler ABI info - done
0.489 -- Check for working C compiler: /usr/bin/cc - skipped
0.489 -- Detecting C compile features
0.490 -- Detecting C compile features - done
0.496 -- Detecting CXX compiler ABI info
0.581 -- Detecting CXX compiler ABI info - done
0.597 -- Check for working CXX compiler: /usr/bin/c++ - skipped
0.597 -- Detecting CXX compile features
0.598 -- Detecting CXX compile features - done
0.600 -- PROJECT VERSION IS: 1.0.5
0.600 Building without ASM
0.615 CMake Error at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
0.615   Could NOT find Python3 (missing: Development Development.Module
0.615   Development.Embed) (Required is exact version "3.10.14")
0.615 Call Stack (most recent call first):
0.615   /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
0.615   /usr/share/cmake-3.25/Modules/FindPython/Support.cmake:3240 (find_package_handle_standard_args)
0.615   /usr/share/cmake-3.25/Modules/FindPython3.cmake:490 (include)
0.615   /app/usd/pxrConfig.cmake:44 (find_package)
0.615   utils/CMakeLists.txt:3 (find_package)
0.615
0.615
0.615 -- Configuring incomplete, errors occurred!
0.615 See also "/app/USD-Fileformat-plugins-1.0.5/build/CMakeFiles/CMakeOutput.log".
issacclee commented 2 weeks ago

Turns out I've made an error when passing -DPython3_LIBRARY parameter to cmake. I originally wrote -DPython3_LIBRARY=/usr/local/lib and cmake find_python3 module would failed. Now I have changed it to -DPython3_LIBRARY=/usr/local/lib/python3.10, which would pass the find_python3 check, but I'm getting a new error halfway through.

 > [12/13] RUN cmake --build build:
0.820 [  3%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/assetresolver.cpp.o
11.71 [  6%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/common.cpp.o
21.46 [  9%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/dictencoder.cpp.o
28.46 [ 12%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/geometry.cpp.o
42.77 [ 15%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/transforms.cpp.o
49.97 [ 18%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/images.cpp.o
61.47 [ 21%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/layerRead.cpp.o
76.88 In file included from /usr/include/c++/12/string:53,
76.88                  from /app/usd/include/pxr/base/tf/hash.h:35,
76.88                  from /app/usd/include/pxr/base/tf/token.h:36,
76.88                  from /app/usd/include/pxr/base/tf/staticTokens.h:75,
76.88                  from /app/USD-Fileformat-plugins-1.0.5/utils/common.h:14,
76.88                  from /app/USD-Fileformat-plugins-1.0.5/utils/usdData.h:24,
76.88                  from /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.h:14,
76.88                  from /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp:12:
76.88 In member function ‘std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::length() const [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’,
76.88     inlined from ‘std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:544:62,
76.88     inlined from ‘pxrInternal_v0_24__pxrReserved__::ArResolvedPath::ArResolvedPath(const std::string&)’ at /app/usd/include/pxr/usd/ar/resolvedPath.h:44:11,
76.88     inlined from ‘bool adobe::usd::readImage(ReadLayerContext&, const pxrInternal_v0_24__pxrReserved__::SdfAssetPath&, int&)’ at /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp:871:55:
76.88 /usr/include/c++/12/bits/basic_string.h:1071:16: warning: dangling pointer to an unnamed temporary may be used [-Wdangling-pointer=]
76.88  1071 |       { return _M_string_length; }
76.88       |                ^~~~~~~~~~~~~~~~
76.88 /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp: In function ‘bool adobe::usd::readImage(ReadLayerContext&, const pxrInternal_v0_24__pxrReserved__::SdfAssetPath&, int&)’:
76.88 /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp:850:69: note: unnamed temporary defined here
76.88   850 |                                    ? PXR_NS::ArGetResolver().Resolve(path.GetAssetPath())
76.88       |                                                                     ^
76.88 In member function ‘std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::pointer std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_M_data() const [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’,
76.88     inlined from ‘std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:544:45,
76.88     inlined from ‘pxrInternal_v0_24__pxrReserved__::ArResolvedPath::ArResolvedPath(const std::string&)’ at /app/usd/include/pxr/usd/ar/resolvedPath.h:44:11,
76.88     inlined from ‘bool adobe::usd::readImage(ReadLayerContext&, const pxrInternal_v0_24__pxrReserved__::SdfAssetPath&, int&)’ at /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp:871:55:
76.88 /usr/include/c++/12/bits/basic_string.h:234:28: warning: dangling pointer to an unnamed temporary may be used [-Wdangling-pointer=]
76.88   234 |       { return _M_dataplus._M_p; }
76.88       |                            ^~~~
76.88 /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp: In function ‘bool adobe::usd::readImage(ReadLayerContext&, const pxrInternal_v0_24__pxrReserved__::SdfAssetPath&, int&)’:
76.88 /app/USD-Fileformat-plugins-1.0.5/utils/layerRead.cpp:850:69: note: unnamed temporary defined here
76.88   850 |                                    ? PXR_NS::ArGetResolver().Resolve(path.GetAssetPath())
76.88       |                                                                     ^
78.35 [ 25%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/layerWriteShared.cpp.o
89.72 [ 28%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/layerWriteMaterial.cpp.o
101.1 [ 31%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/layerWriteMaterialX.cpp.o
115.0 [ 34%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/layerWriteSdfData.cpp.o
131.5 [ 37%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/materials.cpp.o
145.2 [ 40%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/resolver.cpp.o
155.7 [ 43%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/sdfMaterialUtils.cpp.o
172.4 [ 46%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/sdfUtils.cpp.o
183.8 [ 50%] Building CXX object utils/CMakeFiles/fileformatUtils.dir/usdData.cpp.o
191.5 In file included from /app/USD-Fileformat-plugins-1.0.5/utils/common.h:15,
191.5                  from /app/USD-Fileformat-plugins-1.0.5/utils/usdData.h:24,
191.5                  from /app/USD-Fileformat-plugins-1.0.5/utils/usdData.cpp:12:
191.5 /app/USD-Fileformat-plugins-1.0.5/utils/usdData.cpp: In function ‘void adobe::usd::printMaterial(const std::string&, const pxrInternal_v0_24__pxrReserved__::SdfPath&, const Material&, const std::string&)’:
191.5 /app/USD-Fileformat-plugins-1.0.5/utils/usdData.cpp:191:7: warning: too many arguments for format [-Wformat-extra-args]
191.5   191 |       "%s: %s material { %s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s\n",
191.5       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197.1 [ 53%] Linking CXX shared library libfileformatUtils.so
197.4 /usr/bin/ld: cannot find /usr/local/lib/python3.10: file format not recognized
197.4 collect2: error: ld returned 1 exit status
197.4 gmake[2]: *** [utils/CMakeFiles/fileformatUtils.dir/build.make:365: utils/libfileformatUtils.so] Error 1
197.4 gmake[1]: *** [CMakeFiles/Makefile2:151: utils/CMakeFiles/fileformatUtils.dir/all] Error 2
197.4 gmake: *** [Makefile:156: all] Error 2
------
Dockerfile:39
--------------------
  37 |     RUN cmake -S . -B build -DCMAKE_INSTALL_PREFIX=bin -DCMAKE_BUILD_TYPE=Release -Dpxr_ROOT=/app/usd -DUSD_FILEFORMATS_BUILD_TESTS=OFF -DUSD_FILEFORMATS_ENABLE_FBX=OFF -DUSD_FILEFORMATS_ENABLE_GLTF=ON -DUSD_FILEFORMATS_ENABLE_SBSAR=OFF -DUSD_FILEFORMATS_ENABLE_OBJ=OFF -DUSD_FILEFORMATS_ENABLE_PLY=OFF -DUSD_FILEFORMATS_ENABLE_STL=OFF -DUSD_FILEFORMATS_FETCH_GTEST=OFF -DUSD_FILEFORMATS_FETCH_HAPPLY=OFF -DUSD_FILEFORMATS_FETCH_FMT=OFF -DUSD_FILEFORMATS_FETCH_FASTFLOAT=OFF -DUSD_FILEFORMATS_ENABLE_CXX11_ABI=ON -DOpenImageIO_INCLUDE_DIR=/app/usd/include -DOpenImageIO_INCLUDES=/app/usd/include -DOpenImageIO_LIB_DIR=/app/usd/lib -DPython3_INCLUDE_DIR=/usr/local/include/python3.10 -DPython3_LIBRARY=/usr/local/lib/python3.10 -DBoost_INCLUDE_DIR=/usr/local/include/
  38 |
  39 | >>> RUN cmake --build build
  40 |     RUN cmake --install build
  41 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cmake --build build" did not complete successfully: exit code: 2
kwblackstone commented 1 week ago

Hello, I was able to get this dockerfile to build. I ended up creating a symlink for python to ensure it found it where it needed it, along with putting the full path to the python .so here: DPython3_LIBRARY=/usr/local/lib/libpython3.10.so. This also has a couple other minor changes. Let me know if this helps.

  1. Adding python3-dev to the RUN apt-get update
  2. LD_LIBRARY_PATH had "user" instead of "usr" on the last path
FROM python:3.10.14-slim-bookworm
WORKDIR /app

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    ca-certificates \
    git \
    unzip \
    curl \
    cmake \
    build-essential \
    python3-dev

#download the prebuilt usd and place it in workdir
ADD http://demo.uality.cn/usd-2405-ubuntu-22.04.zip /app/usd.zip

RUN unzip /app/usd.zip -d /app/usd

ADD https://github.com/adobe/USD-Fileformat-plugins/archive/refs/tags/v1.0.5.zip /app/release.zip

RUN unzip /app/release.zip -d /app/

WORKDIR /app/USD-Fileformat-plugins-1.0.5

ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/usd/bin:/app/usd/lib:/app/usd/lib64:/app/usd/plugin/usd:/app/USD-Fileformat-plugins-1.0.5/bin/bin:/app/USD-Fileformat-plugins-1.0.5/bin/lib:/app/USD-Fileformat-plugins-1.0.5/bin/plugin/usd:/usr/local/lib
ENV USD_BUILD_PATH=/app/usd
ENV PYTHONPATH=$PYTHONPATH:/app/usd/lib/python
ENV PXR_PLUGINPATH_NAME=/app/USD-Fileformat-plugins-1.0.5/bin/plugin/usd:/app/usd/plugin/usd
ENV PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
ENV Python_ROOT_DIR=/usr/local

# Create symlink to the expected Python include path
RUN mkdir -p /opt/hostedtoolcache/Python/3.10.14/x64/include \
    && ln -s /usr/local/include/python3.10 /opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10

# Build the USD file format plugins
RUN cmake -S . -B build -DCMAKE_INSTALL_PREFIX=bin -DCMAKE_BUILD_TYPE=Release \
    -Dpxr_ROOT=/app/usd \
    -DUSD_FILEFORMATS_BUILD_TESTS=OFF \
    -DUSD_FILEFORMATS_ENABLE_FBX=OFF \
    -DUSD_FILEFORMATS_ENABLE_GLTF=ON \
    -DUSD_FILEFORMATS_ENABLE_SBSAR=OFF \
    -DUSD_FILEFORMATS_ENABLE_OBJ=OFF \
    -DUSD_FILEFORMATS_ENABLE_PLY=OFF \
    -DUSD_FILEFORMATS_ENABLE_STL=OFF \
    -DUSD_FILEFORMATS_FETCH_GTEST=OFF \
    -DUSD_FILEFORMATS_FETCH_HAPPLY=OFF \
    -DUSD_FILEFORMATS_FETCH_FMT=OFF \
    -DUSD_FILEFORMATS_FETCH_FASTFLOAT=OFF \
    -DUSD_FILEFORMATS_ENABLE_CXX11_ABI=ON \
    -DOpenImageIO_INCLUDE_DIR=/app/usd/include \
    -DOpenImageIO_INCLUDES=/app/usd/include \
    -DOpenImageIO_LIB_DIR=/app/usd/lib \
    -DPython3_INCLUDE_DIR=/usr/local/include/python3.10 \
    -DPython3_LIBRARY=/usr/local/lib/libpython3.10.so

RUN cmake --build build
RUN cmake --install build

ENTRYPOINT [ "/app/usd/bin/usdcat" ]
issacclee commented 1 week ago

ENTRYPOINT [ "/app/usd/bin/usdcat" ]

It builds just fine, thank you so much. I'll close the issue for now.

issacclee commented 1 week ago

Hi, I ran the container with following command

docker run --rm -v C:/Users/UalityMobWorkstation/Downloads:/app uality-usd-fileplugin:latest -o test.usd /app/bb25fevi_ld2_sceneViewer.glb

An error prompt out saying:

Failed to open "/app/bb25fevi_ld2_sceneViewer.glb" - Cannot determine file format for @/app/bb25fevi_ld2_sceneViewer.glb@

Does that mean the environment variable for adobe-fileformat-plugin didn't actually get setup properly? The glb file is definitely valid. bb25fevi_ld2_sceneViewer.zip

kwblackstone commented 6 days ago

That error typically means PXR_PLUGINPATH_NAME wasn't set. You'll want to make sure all three of these are setup properly: https://github.com/adobe/USD-Fileformat-plugins?tab=readme-ov-file#for-linuxmac-1

issacclee commented 6 days ago

That error typically means PXR_PLUGINPATH_NAME wasn't set. You'll want to make sure all three of these are setup properly: adobe/USD-Fileformat-plugins#for-linuxmac-1

As the link suggested, we should either copy all the contents of USD-Fileformat-plugins/bin/bin and USD-Fileformat-plugins/bin/plugin/usd to the {USD_Install_Path}/plugin/usd or set PATH and PXR_PLUGINPATH_NAME accordingly.

However when I inspect the bin folder in usd-fileformat-plugin install path, I found that there actually is no /bin/bin folder image

Does this mean the installation process of usd-fileformat-plugin didn't actually succeed?

kwblackstone commented 9 hours ago

interesting, maybe... the fileformatUtils shared lib to be there. Can you check if the build\utils\Release folder contains it? If not maybe it failed to build properly.

issacclee commented 8 hours ago

image

There seems to be no Release folder in build\utils\Release