search

Project-OSRM / osrm-backend

Open Source Routing Machine - C++ backend
http://map.project-osrm.org
BSD 2-Clause "Simplified" License
6.21k stars 3.29k forks source link

osrm-extract PBFParser exception causes program to hang #4682

Open shacker27 opened 6 years ago

shacker27 commented 6 years ago

I'm trying to build and run osrm-extract (v5.13.0) on CentOS 6.9 and have followed these instructions to build the package. The only exception is that I am building OSRM in debug mode so I include symbols.

When attempting to extract the example Berlin pbf via /usr/local/bin/osrm-extract -p /usr/local/share/osrm/profiles/car.lua berlin-latest.osm.pbf I see several lines of log output, up until Parse relations ... (seems like this line), at which point the program just hangs forever.

Running the program in gdb and telling gdb to catch exceptions, I get the following stack trace (Note: Program log output included for context):

Starting program: /usr/local/bin/osrm-extract -p /usr/local/share/osrm/profiles/car.lua berlin-latest.osm.pbf
[Thread debugging using libthread_db enabled]
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /usr/local/share/osrm/profiles/car.lua
[info] Input file: district-of-columbia-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[New Thread 0x7ffff61cc700 (LWP 70909)]
[New Thread 0x7ffff41cb700 (LWP 70910)]
[New Thread 0x7ffff21ca700 (LWP 70911)]
[New Thread 0x7ffff01c9700 (LWP 70912)]
[info] Parsing in progress..
[New Thread 0x7fffee1c8700 (LWP 70913)]
[New Thread 0x7fffe7fff700 (LWP 70914)]
[New Thread 0x7fffe5ffe700 (LWP 70915)]
[New Thread 0x7fffe3ffd700 (LWP 70916)]
[info] input file generated by osmium/1.5.1
[info] timestamp: 2017-11-12T21:43:02Z
[Thread 0x7fffee1c8700 (LWP 70913) exited]
[Thread 0x7fffe3ffd700 (LWP 70916) exited]
[info] Using profile api version 3
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
[New Thread 0x7fffe3ffd700 (LWP 70917)]
[New Thread 0x7fffee1c8700 (LWP 70918)]
[Thread 0x7fffe3ffd700 (LWP 70917) exited]
[Switching to Thread 0x7fffee1c8700 (LWP 70918)]
Catchpoint 1 (exception thrown), 0x00007ffff7522cb0 in __cxa_throw () from /usr/lib64/libstdc++.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.209.el6_9.2.x86_64 libgcc-4.4.7-18.el6.x86_64 libgomp-4.4.7-18.el6.x86_64 libstdc++-4.4.7-18.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff7522cb0 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#1  0x00000000010a0376 in osmium::io::detail::PBFParser::read_from_input_queue (this=0x1672f50, size=4) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/pbf_input_format.hpp:80
#2  0x00000000010a0444 in osmium::io::detail::PBFParser::read_blob_header_size_from_file (this=Unhandled dwarf expression opcode 0xf3
) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/pbf_input_format.hpp:102
#3  0x00000000010a04f5 in osmium::io::detail::PBFParser::check_type_and_get_blob_size (this=0x1672f50, expected_type=0x138f87a "OSMData") at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/pbf_input_format.hpp:151
#4  0x00000000010bd286 in osmium::io::detail::PBFParser::parse_data_blobs (this=0x1672f50) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/pbf_input_format.hpp:177
#5  0x00000000010bd393 in osmium::io::detail::PBFParser::run (this=0x1672f50) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/pbf_input_format.hpp:205
#6  0x00000000010b6bc8 in osmium::io::detail::Parser::parse (this=0x1672f50) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/detail/input_format.hpp:153
#7  0x00000000010b6cb8 in osmium::io::Reader::parser_thread (pool=..., creator=..., input_queue=..., osmdata_queue=..., header_promise=Unhandled dwarf expression opcode 0xf3
) at /opt/osrm-backend-5.13.0/third_party/libosmium/include/osmium/io/reader.hpp:157
#8  0x00000000010c23bb in std::_Bind_simple<void (*(std::reference_wrapper<osmium::thread::Pool>, std::reference_wrapper<std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)> >, std::reference_wrapper<osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::reference_wrapper<osmium::thread::Queue<std::future<osmium::memory::Buffer> > >, std::promise<osmium::io::Header>, osmium::osm_entity_bits::type, osmium::io::read_meta))(osmium::thread::Pool&, const std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)>&, osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, osmium::thread::Queue<std::future<osmium::memory::Buffer> >&, std::promise<osmium::io::Header>&&, osmium::osm_entity_bits::type, osmium::io::read_meta)>::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>) (this=0x1673018) at /opt/rh/devtoolset-6/root/usr/include/c++/6.3.1/functional:1391
#9  0x00000000010c23dd in std::_Bind_simple<void (*(std::reference_wrapper<osmium::thread::Pool>, std::reference_wrapper<std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)> >, std::reference_wrapper<osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::reference_wrapper<osmium::thread::Queue<std::future<osmium::memory::Buffer> > >, std::promise<osmium::io::Header>, osmium::osm_entity_bits::type, osmium::io::read_meta))(osmium::thread::Pool&, const std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)>&, osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, osmium::thread::Queue<std::future<osmium::memory::Buffer> >&, std::promise<osmium::io::Header>&&, osmium::osm_entity_bits::type, osmium::io::read_meta)>::operator()(void) (
    this=Unhandled dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-6/root/usr/include/c++/6.3.1/functional:1380
#10 0x00000000010c23f1 in std::thread::_State_impl<std::_Bind_simple<void (*(std::reference_wrapper<osmium::thread::Pool>, std::reference_wrapper<std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)> >, std::reference_wrapper<osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::reference_wrapper<osmium::thread::Queue<std::future<osmium::memory::Buffer> > >, std::promise<osmium::io::Header>, osmium::osm_entity_bits::type, osmium::io::read_meta))(osmium::thread::Pool&, const std::function<std::unique_ptr<osmium::io::detail::Parser, std::default_delete<osmium::io::detail::Parser> >(osmium::io::detail::parser_arguments&)>&, osmium::thread::Queue<std::future<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, osmium::thread::Queue<std::future<osmium::memory::Buffer> >&, std::promise<osmium::io::Header>&&, osmium::osm_entity_bits::type, osmium::io::read_meta)> >::_M_run(void) (this=Unhandled dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-6/root/usr/include/c++/6.3.1/thread:197
#11 0x000000000137f3ff in execute_native_thread_routine ()
#12 0x00007ffff7bc5c5a in start_thread (arg=0x7fffee1c8700) at pthread_create.c:301
#13 0x00007ffff6b03fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

I've built my glibc-2.14 installation in /opt/glibc-2.14 and you can see what osrm-extract is linking to with the output of ldd below:

ldd /usr/local/bin/osrm-extract
  linux-vdso.so.1 =>  (0x00007ffc441c7000)
  libpthread.so.0 => /opt/glibc-2.14/lib/libpthread.so.0 (0x00007fd192451000)
  libtbb.so.2 => /opt/osrm-backend-5.13.0/mason_packages/linux-x86_64/tbb/2017_U7/lib/libtbb.so.2 (0x00007fd192213000)
  libz.so.1 => /lib64/libz.so.1 (0x00007fd191ff0000)
  libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fd191cea000)
  libm.so.6 => /opt/glibc-2.14/lib/libm.so.6 (0x00007fd191a66000)
  libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007fd191851000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd19163b000)
  libc.so.6 => /opt/glibc-2.14/lib/libc.so.6 (0x00007fd1912ae000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fd19266f000)
  libdl.so.2 => /opt/glibc-2.14/lib/libdl.so.2 (0x00007fd1910aa000)
  librt.so.1 => /opt/glibc-2.14/lib/librt.so.1 (0x00007fd190e8f000)
oxidase commented 6 years ago

@shacker27 i reproduced the issue as

docker run -v /home/miha/mapbox/osrm-backend:/osrm-host --security-opt seccomp=unconfined -it centos:6.9 /bin/bash
yum install yum-utils centos-release-scl
yum-config-manager --enable rhel-server-rhscl-6-rpms
yum install devtoolset-6
yum install git cmake zlib-devel wget autoconf automake
scl enable devtoolset-6 bash

cd
wget http://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.gz
tar xvzf glibc-2.14.tar.gz
cd glibc-2.14 && mkdir build && cd build
sed -i 's/3\.4\* | 4\.\[0-9\]\* )/3.4* | 4.[0-9]* | 6.* )/g' ../configure
sed -i 's/3\.79\* | 3\.\[89\]\*)/3.79* | 3.[89]* | 4.* )/g' ../configure
../configure --prefix=/opt/glibc-2.14
make -j4
make install

cd
git clone https://github.com/Project-OSRM/osrm-backend.git osrm-backend
mkdir osrm-backend/build
cd osrm-backend/build
cmake .. -DENABLE_MASON=ON -DCMAKE_BUILD_TYPE=Debug
make -j4

wget http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf
export LD_LIBRARY_PATH=/opt/glibc-2.14/lib
./osrm-extract -p ../profiles/car.lua berlin-latest.osm.pbf

The problem here is in TBB installed by mason.

After a change of

  mason_use(tbb VERSION ${MASON_TBB_VERSION})
  add_dependency_includes(${MASON_PACKAGE_tbb_INCLUDE_DIRS})
  set(TBB_LIBRARIES ${MASON_PACKAGE_tbb_LDFLAGS})

in CMakeLists.txt to

  find_package(TBB REQUIRED)
  add_dependency_includes(${TBB_INCLUDE_DIR})
  if(WIN32 AND CMAKE_BUILD_TYPE MATCHES Debug)
    set(TBB_LIBRARIES ${TBB_DEBUG_LIBRARIES})
  endif()

and making a local TBB build

cd
wget https://github.com/01org/tbb/archive/2018_U1.zip
unzip 2018_U1.zip
cd tbb-2018_U1
make -j4

cd /root/osrm-backend/build
export LD_LIBRARY_PATH=/opt/glibc-2.14/lib:/root/tbb-2018_U1/build/linux_intel64_gcc_cc6.3.1_libc2.14_kernel4.10.0_release
cmake .. -DENABLE_MASON=ON -DCMAKE_BUILD_TYPE=Debug -DTBB_INSTALL_DIR=/root/tbb-2018_U1
./osrm-extract -p ../profiles/car.lua berlin-latest.osm.pbf

osrm-extract works without hanging at TBB code.

shacker27 commented 6 years ago

Hey @oxidase thanks for verifying this! I've been battling it for a while so incredibly helpful to see what the issue is.

When I change CMakeLists.txt as specified above and add the -DTBB_INSTALL_DIR to the cmake command, I get the following error:

CMake Error at CMakeLists.txt:470 (message):
  MASON_PACKAGE_tbb_LIBRARY_DIRS is empty, rpath will not work

I am by no means a cmake expert but commenting out 

if(NOT MASON_PACKAGE_tbb_LIBRARY_DIRS)
  message(FATAL_ERROR "MASON_PACKAGE_tbb_LIBRARY_DIRS is empty, rpath will not work")
endif()

allows me to build and things seem to be working.

github-actions[bot] commented 3 weeks ago

This issue seems to be stale. It will be closed in 30 days if no further activity occurs.