search

gazebosim / sdformat

Simulation Description Format (SDFormat) parser and description files.
http://sdformat.org
Apache License 2.0
165 stars 94 forks source link

Improve documentation and console messaging about USE_INTERNAL_URDF cmake option #1124

Open julianoes opened 2 years ago

julianoes commented 2 years ago

Environment

Description

I'm trying to build Ignition Gazebo 6 on Debian. The colcon build fails with a linking error for sdformat12. It looks like something goes wrong with the linking of tinyxml2 symbols. Presumably the tinyxml2 library installed on my system conflicts with tinyxml2 in tree in gz-common.

I tried to build everything without having tinyxml2 installed system-wide but that does not work either as it is required by sdformat12.

Steps to reproduce

vcs import < collection-fortress.yaml
colcon graph
colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install

Should I provide a debian based Dockerfile to reproduce it?

Output

Colcon build without libtinyxml2-dev installed system-wide:

Starting >>> sdformat12                                                              
--- stderr: sdformat12                                                 
CMake Error at /home/julianoes/src/other/ign-workspace/install/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:59 (message):
  -- BUILD ERRORS: These must be resolved before compiling.
Call Stack (most recent call first):
  CMakeLists.txt:115 (ign_configure_build)

CMake Error at /home/julianoes/src/other/ign-workspace/install/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:61 (message):
  --    Missing dependency [TINYXML2]
Call Stack (most recent call first):
  CMakeLists.txt:115 (ign_configure_build)

Build with tinyxml2 installed:

/usr/bin/ld: CMakeFiles/sdformat12.dir/urdf/urdf_parser/model.cpp.o: in function `tinyxml2::XMLDocument::ClearError()':
/usr/include/tinyxml2.h:1871: undefined reference to `tinyxml2::XMLDocument::SetError(tinyxml2::XMLError, int, char const*, ...)'
/usr/bin/ld: ../lib/libsdformat12.so.12.5.0: hidden symbol `_ZN8tinyxml211XMLDocument8SetErrorENS_8XMLErrorEiPKcz' isn't defined
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/sdformat12.dir/build.make:1247: lib/libsdformat12.so.12.5.0] Error 1

I've initially asked in https://github.com/gazebosim/gz-common/issues/430 but I'm adding the issue here as well, as it's sdformat failing to link.

scpeters commented 2 years ago

Should I provide a debian based Dockerfile to reproduce it?

yes please, that would be helpful.

What is the version of tinyxml2 installed in your system?

julianoes commented 2 years ago

The version should be: https://packages.debian.org/bullseye/libtinyxml2-8

julianoes commented 2 years ago

I tried to create a Dockerfile to reproduce but I got stuck earlier. I'll post it here, in case anyone knows how to fix it.

This is the Dockerfile:

FROM debian:bullseye

RUN apt-get update \
    && apt dist-upgrade -y \
    && apt install -y \
        git \
        wget \
        python3-pip \
        python-is-python3 \
        python3-distutils \
        build-essential \
        gfortran \
        cmake

RUN python -m pip install vcstool
RUN python -m pip install colcon-common-extensions

RUN wget https://raw.githubusercontent.com/ignition-tooling/gazebodistro/master/collection-fortress.yaml
RUN vcs import < collection-fortress.yaml
RUN colcon graph
RUN colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install

Then I run it with docker build . -f Dockerfile, and I get the error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/distutils/core.py", line 225, in run_setup
    raise RuntimeError(("'distutils.core.setup()' was never called -- "
RuntimeError: 'distutils.core.setup()' was never called -- perhaps 'setup.py' is not a Distutils setup script?
[13.014s] ERROR:colcon.colcon_core.package_identification:Exception in package identification extension 'python_setup_py' in 'usr/lib/python3.9/test/libregrtest': Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/colcon_core/package_identification/__init__.py", line 142, in _identify
    retval = extension.identify(_reused_descriptor_instance)
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 48, in identify
    config = get_setup_information(setup_py)
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 241, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 281, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.

FortranCInterface     +                   
IntelFortranImplicit   +                  
git                     +                 
ignition-cmake2          +  **************
ignition-tools            +             **
my-test-package            +              
ignition-math6              + *****.******
ignition-utils1              +* *..**...**
ignition-common4              +  ** ******
ignition-msgs8                 + * *  ****
sdformat12                      +   *  **.
ignition-fuel_tools7             +      *.
ignition-plugin1                  + ***.**
ignition-transport11               +  ****
ignition-physics5                   +   *.
ignition-rendering6                  +***.
ignition-gui6                         + **
ignition-sensors6                      +*.
ignition-gazebo6                        +*
ignition-launch5                         +
Removing intermediate container 3a9c5f2e79b3
 ---> c100eddaeb58
Step 8/8 : RUN colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install
 ---> Running in bdf4ad761cdc
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/distutils/core.py", line 225, in run_setup
    raise RuntimeError(("'distutils.core.setup()' was never called -- "
RuntimeError: 'distutils.core.setup()' was never called -- perhaps 'setup.py' is not a Distutils setup script?
[13.111s] ERROR:colcon.colcon_core.package_identification:Exception in package identification extension 'python_setup_py' in 'usr/lib/python3.9/test/libregrtest': Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/colcon_core/package_identification/__init__.py", line 142, in _identify
    retval = extension.identify(_reused_descriptor_instance)
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 48, in identify
    config = get_setup_information(setup_py)
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 241, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/usr/local/lib/python3.9/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 281, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.

Starting >>> ignition-cmake2
Starting >>> ignition-tools
Starting >>> FortranCInterface
Starting >>> IntelFortranImplicit
Starting >>> git
Starting >>> my-test-package
--- stderr: git
Generating GIT-VERSION-FILE
/bin/sh: 0: cannot open /usr/share/doc/git/contrib/buildsystems/../../GIT-VERSION-GEN: No such file
CMake Error at CMakeLists.txt:77 (file):
  file STRINGS file
  "/usr/share/doc/git/contrib/buildsystems/../../GIT-VERSION-FILE" cannot be
  read.

CMake Error at CMakeLists.txt:78 (string):
  string sub-command REPLACE requires at least four arguments.

CMake Error at CMakeLists.txt:79 (string):
  string sub-command FIND requires 3 or 4 parameters.

CMake Error at CMakeLists.txt:83 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.

CMake Error at CMakeLists.txt:87 (project):
  VERSION ".0" format invalid.

---
Failed   <<< git [0.15s, exited with code 1]
Aborted  <<< IntelFortranImplicit [0.23s]
Aborted  <<< ignition-tools [0.31s]
Aborted  <<< ignition-cmake2 [0.37s]
Aborted  <<< my-test-package [0.32s]
Aborted  <<< FortranCInterface [0.37s]

Summary: 0 packages finished [14.7s]
  1 package failed: git
  5 packages aborted: FortranCInterface IntelFortranImplicit ignition-cmake2 ignition-tools my-test-package
  3 packages had stderr output: FortranCInterface IntelFortranImplicit git
  14 packages not processed
The command '/bin/sh -c colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install' returned a non-zero code: 1
scpeters commented 2 years ago

I tried to create a Dockerfile to reproduce but I got stuck earlier. I'll post it here, in case anyone knows how to fix it.

it looks like you are calling vcs checkout from the the default shell path, which I believe is /. I recommend creating a dedicated workspace folder (~/ws or /ws for a root user docker container) and vcs importing the source code into ws/src

FortranCInterface     +                   
IntelFortranImplicit   +                  
git                     +

those should not be in the colcon graph

julianoes commented 2 years ago

Thanks for the hint @scpeters. With a proper directory, I was able to reproduce it:

FROM debian:bullseye

RUN apt-get update \
    && apt dist-upgrade -y \
    && apt install -y \
        git \
        wget \
        python3-pip \
        python-is-python3 \
        python3-distutils \
        build-essential \
        cmake \
        libtinyxml2-dev \
        libavutil-dev \
        libavcodec-dev \
        libavformat-dev \
        libavdevice-dev \
        libfreeimage-dev \
        libeigen3-dev \
        libgts-dev \
        libprotobuf-dev \
        libprotoc-dev \
        protobuf-compiler \
        protobuf-c-compiler \
        pkg-config \
        ruby

RUN python -m pip install vcstool
RUN python -m pip install colcon-common-extensions

RUN mkdir -p /ws/src
WORKDIR /ws/src
RUN wget https://raw.githubusercontent.com/ignition-tooling/gazebodistro/master/collection-fortress.yaml
RUN vcs import < collection-fortress.yaml
RUN colcon graph
RUN colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install

The output is then:

...
Step 10/10 : RUN colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install
 ---> Running in 5daaba0e273e
Starting >>> ignition-cmake2
Starting >>> ignition-tools
Finished <<< ignition-cmake2 [0.41s]
Starting >>> ignition-math6
Starting >>> ignition-utils1
Finished <<< ignition-tools [1.01s]
Finished <<< ignition-utils1 [1.65s]
--- stderr: ignition-math6
CMake Warning at /ws/src/install/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:55 (message):
   CONFIGURATION WARNINGS:
   -- Swig is missing: Language interfaces are disabled.
   -- pybind11 is missing: Python interfaces are disabled.
Call Stack (most recent call first):
  CMakeLists.txt:104 (ign_configure_build)

---
Finished <<< ignition-math6 [7.31s]
Starting >>> ignition-common4
Starting >>> ignition-msgs8
Starting >>> sdformat12
[Processing: ignition-common4, ignition-msgs8, sdformat12]
Finished <<< ignition-common4 [51.8s]
Starting >>> ignition-plugin1
Finished <<< ignition-plugin1 [7.29s]
Starting >>> ignition-rendering6
--- stderr: sdformat12
CMake Warning at /ws/src/install/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:55 (message):
   CONFIGURATION WARNINGS:
   -- Python psutil package not found. Memory leak tests will be skipped
Call Stack (most recent call first):
  CMakeLists.txt:115 (ign_configure_build)

/ws/src/sdformat/src/urdf/urdf_parser/world.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/world.cpp:72: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   72 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/urdf_sensor.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/urdf_sensor.cpp:395: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  395 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/urdf_model_state.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/urdf_model_state.cpp:166: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  166 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/twist.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/twist.cpp:81: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   81 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/pose.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/pose.cpp:133: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  133 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/joint.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/joint.cpp:694: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  694 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/link.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/link.cpp:679: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  679 | #pragma warning(pop)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/model.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      | 
/ws/src/sdformat/src/urdf/urdf_parser/model.cpp:264: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
  264 | #pragma warning(pop)
      | 
/usr/bin/ld: CMakeFiles/sdformat12.dir/urdf/urdf_parser/model.cpp.o: in function `tinyxml2::XMLDocument::ClearError()':
/usr/include/tinyxml2.h:1871: undefined reference to `tinyxml2::XMLDocument::SetError(tinyxml2::XMLError, int, char const*, ...)'
/usr/bin/ld: ../lib/libsdformat12.so.12.6.0~pre1: hidden symbol `_ZN8tinyxml211XMLDocument8SetErrorENS_8XMLErrorEiPKcz' isn't defined
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/sdformat12.dir/build.make:1247: lib/libsdformat12.so.12.6.0~pre1] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:451: src/CMakeFiles/sdformat12.dir/all] Error 2
gmake: *** [Makefile:182: all] Error 2
---
Failed   <<< sdformat12 [1min 4s, exited with code 2]
Aborted  <<< ignition-rendering6 [19.1s]
Aborted  <<< ignition-msgs8 [1min 21s]

Summary: 6 packages finished [1min 29s]
  1 package failed: sdformat12
  2 packages aborted: ignition-msgs8 ignition-rendering6
  3 packages had stderr output: ignition-math6 ignition-rendering6 sdformat12
  7 packages not processed
The command '/bin/sh -c colcon build --cmake-args -DBUILD_TESTING=OFF --merge-install' returned a non-zero code: 2
scpeters commented 2 years ago

I see a clue in the pragma warnings:

/ws/src/sdformat/src/urdf/urdf_parser/world.cpp:36: warning: ignoring ‘#pragma warning ’ [-Wunknown-pragmas]
   36 | #pragma warning(push, 0)
      |

libsdformat has a vendor version of urdfdom and urdfdom_headers in the src/urdf folder, which is where this compiler warning is coming from. If liburdfdom-dev is installed, then I think it would not have this problem.

This is still a problem, but can you try installing liburdfdom-dev, wiping the build folder and trying again?

julianoes commented 2 years ago

Thanks @scpeters. I think installing liburdfdom-dev seemed to fix that part of the build indeed. I assume there should be some configure step that makes sure the system one is available, so that this problem doesn't occur?

The build now fails due to a similar problem than https://github.com/gazebosim/gz-transport/pull/237 but I'll have to create a separate issue for that.

julianoes commented 2 years ago

Re gz-transport build, follow up in https://github.com/gazebosim/gz-transport/issues/362 and https://github.com/gazebosim/gz-transport/pull/363.

scpeters commented 2 years ago

Thanks @scpeters. I think installing liburdfdom-dev seemed to fix that part of the build indeed. I assume there should be some configure step that makes sure the system one is available, so that this problem doesn't occur?

you can set the cmake variable -DUSE_INTERNAL_URDF=OFF to force using the system installation of urdfdom. Otherwise it uses the vendor version if the external version is not found

julianoes commented 2 years ago

Ok, that's good to know but I just have the feeling this can really confuse one (like me) thoroughly and there has to be a better way to prevent that sort of trouble, no?

azeey commented 1 year ago

What's the state of this issue? Is there an action item?

scpeters commented 1 year ago

What's the state of this issue? Is there an action item?

if anything, I think it is to improve the documentation and console messaging about the -DUSE_INTERNAL_URDF=OFF variable and whether the system version of urdfdom is being used or the from-source fork