search

mapnik / python-mapnik

Python bindings for mapnik
GNU Lesser General Public License v2.1
157 stars 90 forks source link

Python 3.10.4 + boost.Python #258

Closed artemp closed 1 year ago

artemp commented 2 years ago

When built in Release mode import mapnik is causing a segfault (Ubuntu 22.04 + clang 14). Debug build is OK.

artemp commented 1 year ago

clang++-13 on Ubuntu 22.04 works.

artemp commented 1 year ago

llvm-project - https://github.com/llvm/llvm-project/commits/db818219c5b3f81625edff19f2ec4ab166ffcc3f built from source on Ubuntu 22.04 e.g.

cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" -DCMAKE_INSTALL_PREFIX=/opt/llvm -DCMAKE_BUILD_TYPE=Release

NOTE: BOOST_VERSION <= 1.80.0 has an issue with latest clang++ - https://github.com/boostorg/mpl/issues/69

comiquita commented 1 year ago

Hi @artemp, I'm currently facing this issue "segmentation fault (core dumped)". On Ubuntu 22.04, I built boost (boost 1.80.0) and mapnik from source.

When running (for example) demo.py from terminal, I get the above error. When I run it from an IDE in debug mode, it works and no error message.

Will I need to generate maps using the IDE in debug mode? is there another alternative?

artemp commented 1 year ago

Have you tried fix mentioned above dabrahams/mpl@453e0cf ?

comiquita commented 1 year ago

Hi @artemp , just to let you know that I'm still working on this.

I downloaded the latest llvm-project and built and installed.

I applied the change in https://github.com/dabrahams/mpl/commit/453e0cf

Now I'm building boost 1.80.0 with the new compiler but I'm getting the following error:

select_stdlib_config.hpp:26:14: fatal error: 'cstddef' file not found '# include

1 error generated.

So I'm debugging this at the moment

comiquita commented 1 year ago

hi @artemp,

I managed to build boost with latest clang (installed llvm the same way you did), however, I had to revert that fix [https://github.com/dabrahams/mpl/commit/453e0cf ](meaning, I didn't use the fix).

Now, I have the following issue, which I'm not sure if it's related to this discussion.

When I configure mapnik, I get Segmentation Fault (core dumped) after sqlite3 and cairo. Here:

_Checking for C library tiff... yes Checking for PROJ_LIB directory...proj_info.searchpath returned /usr/local/share/proj Checking for ICU data directory...icu-config returned /usr/local/share/icu/71.1 Checking for GDAL data directory... gdal-config returned /usr/share/gdal Checking for requested plugins dependencies... Checking for pg_config... yes Checking for pg_config... yes Checking for gdal-config --libs... yes Checking for gdal-config --cflags... yes Checking for name of gdal library... gdal Checking for C++ library gdal... yes Checking if gdal is ogr enabled... yes Checking for gdal-config --libs... yes Checking for gdal-config --cflags... yes Checking for name of ogr library... gdal Checking for C++ library gdal... yes Checking for C library sqlite3... yes Segmentation fault (core dumped) Checking if SQLite supports RTREE... no Checking for cairo... yes Checking for cairo lib and include paths... yes Segmentation fault (core dumped) Checking for cairo freetype font support ... no

All Required dependencies found!

Overwriting and re-saving file 'config.py'... Will hold custom path variables from commandline and python config file(s)...

Note: will build without these OPTIONAL dependencies:

I never had this before when I had mapnik installed before (maybe I should've mentioned a little background. I used to have a tile server on a mac, but I've migrated all my projects to ubuntu 22.04 and since then I haven't been able to obtain the same condifuration in ubuntu. Therefore, I'm familiar with mapnikm and how it all works, but have issues installing it)

Could you shed some light? as I'm running out of ideas.

Best regards,

Alex.

artemp commented 1 year ago

@comiquita - it's not clear what compiler is used at configure stage, could you post complete stderr? Also, you don't need to build llvm/clang++ from source you can use clang or gcc that comes with Ubuntu 22.04

comiquita commented 1 year ago

@artemp, I had the latest clang installed from source, but thanks for letting me know. I could've just used the compiler that came with 22.04. Also, I built boost with clang as the toolset (to avoid the segfault issue).

Here's the complete results from configure: $ ./configure CXX="clang++" CXX_STD="17" CC="clang" CUSTOM_CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" XMLPARSER=libxml2 DEBUG=True PROJ=True CAIRO=True CUSTOM_DEFINES="-DHAVE_LIBXML2 -DHAVE_CAIRO -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1 -DHAVE_JPEG -DMAPNIK_USE_PROJ -DHAVE_PNG -DHAVE_TIFF -DBOOST_REGEX_HAS_ICU" INPUT_PLUGINS=all NIK2IMG=False --prefix=/opt/mapnik_deps --with-icu=yes --without-iconv --with-threads=yes XML2_INCLUDES=/usr/local/include/libxml XML2_LIBS=/usr/local/lib PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig/

scons: Reading SConscript files ...

Welcome to Mapnik...

Configuring build environment... SCons CONFIG found: 'config.py', variables will be inherited... Configuring on Linux in debug mode... C++ compiler: clang version 16.0.0 (https://github.com/llvm/llvm-project.git 93f1b48cbbeae0a04be560a98daa16350a3da069) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/llvm/bin Checking for pkg-config... yes Checking for freetype2... yes Checking for dlfcn.h support ... yes Checking if compiler (clang++) supports -std=c++17 flag... yes Checking for C library z... yes Checking for C++ library icuuc... yes Checking for ICU version >= 4.0... yes (found ICU 71.1) Checking for C++ library harfbuzz... yes Checking for HarfBuzz version >= 0.9.34... yes (found HarfBuzz 2.7.4) Checking for HarfBuzz with freetype support... yes Searching for boost libs and headers... (cached) Found boost libs: /usr/local/lib Found boost headers: /usr/local/include Checking for C++ header file boost/version.hpp... yes Checking for Boost version >= 1.73... yes Found boost lib version... 1_80 Checking for C++ library boost_system... yes Checking for C++ library boost_filesystem... yes Checking for C++ library boost_regex... yes Checking for C++ library boost_program_options... yes Checking whether Boost was compiled with C++11 scoped enums ... yes Checking if boost_regex was built with ICU unicode support... yes Checking for C library libxml2... yes Checking for C library jpeg... yes Checking for C library proj... yes Checking for Proj version >=7.2.0...yes (found Proj 9.0.1) Checking for C library png... yes Checking for C library webp... yes Checking for C library tiff... yes Checking for PROJ_LIB directory...proj_info.searchpath returned /usr/local/share/proj Checking for ICU data directory...icu-config returned /usr/local/share/icu/71.1 Checking for GDAL data directory... gdal-config returned /usr/share/gdal Checking for requested plugins dependencies... Checking for pg_config... yes Checking for pg_config... yes Checking for gdal-config --libs... yes Checking for gdal-config --cflags... yes Checking for name of gdal library... gdal Checking for C++ library gdal... yes Checking if gdal is ogr enabled... yes Checking for gdal-config --libs... yes Checking for gdal-config --cflags... yes Checking for name of ogr library... gdal Checking for C++ library gdal... yes Checking for C library sqlite3... yes Segmentation fault (core dumped) Checking if SQLite supports RTREE... no Checking for cairo... yes Checking for cairo lib and include paths... yes Segmentation fault (core dumped) Checking for cairo freetype font support ... no

All Required dependencies found!

Overwriting and re-saving file 'config.py'... Will hold custom path variables from commandline and python config file(s)...

Note: will build without these OPTIONAL dependencies:

Configure completed: run make to build or make install

I also tried to configure with clang that came with ubuntu 22.04 and got the same issue: scons: Reading SConscript files ...

Welcome to Mapnik...

Configuring build environment... SCons CONFIG found: 'config.py', variables will be inherited... Configuring on Linux in debug mode... C++ compiler: Ubuntu clang version 14.0.0-1ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

...

Checking for C library sqlite3... yes Segmentation fault (core dumped) Checking if SQLite supports RTREE... no Checking for cairo... yes Checking for cairo lib and include paths... yes Segmentation fault (core dumped) Checking for cairo freetype font support ... no

Note, when I configure with gcc, I don't get the "Segmentation fault (core dumped)" issue. However, since boost was compiled with clang, I need to stick to clang for mapnik, am I correct?

artemp commented 1 year ago

@comiquita Most likely you have multiple versions of various libraries installed which are clashing but I'm only guessing.. Here is what is working for me using boost 1_80 compiled from source and system libraries on Ubuntu 22.04

export LD_LIBRARY_PATH=/opt/boost_1_80/lib:/opt/mapnik/lib

./scons/scons.py  CXX_STD=17 PREFIX=/opt/mapnik
BOOST_INCLUDES=/opt/boost_1_80/include
BOOST_LIBS=/opt/boost_1_80/lib
SVG_RENDERER=True SVG2PNG=True

and I'm using stock clang which comes with 22.04

c++ --version
Ubuntu clang version 14.0.0-1ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

You can use g++ the key is to ensure c++ ABI are matching e.g both clang and gcc use the same runtime e.g libstdc++ . How did you build boost? You can check what boost libs are linked to - 

ldd /opt/boost_1_80/lib/libboost_regex.so
    linux-vdso.so.1 (0x00007ffc094a7000)
    libicudata.so.70 => /lib/x86_64-linux-gnu/libicudata.so.70 (0x00007fe1a2b04000)
    libicui18n.so.70 => /lib/x86_64-linux-gnu/libicui18n.so.70 (0x00007fe1a27d5000)
    libicuuc.so.70 => /lib/x86_64-linux-gnu/libicuuc.so.70 (0x00007fe1a25da000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe1a23b0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe1a22c9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe1a22a7000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe1a207f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe1a4792000)
comiquita commented 1 year ago

Thanks for the replies btw.

This is how I build boost:

sudo ./bootstrap.sh --with-libraries=all --with-toolset=clang define=_GLIBCXX_USE_CXX11_ABI=0 -with-icu=/usr/local --with-python=PYTHON3 link=shared variant=debug threading=multi runtime-link=shared

sudo ./b2 install toolset=clang cxxflags="-std=c++17 -stdlib=libstdc++" define=_GLIBCXX_USE_CXX11_ABI=0 - -sHAVE_ICU=1 link=shared link=shared variant=debug threading=multi runtime-link=shared

Here's my results:

$ ldd /usr/local/lib/libboost_regex.so
    linux-vdso.so.1 (0x00007ffffcd86000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f733d8a5000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f733d7be000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f733d79e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f733d576000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f733db72000)

It looks different to yours, I don't see boost regex to be built with ICU. How did you compile boost from source? could you share?

artemp commented 1 year ago

@comiquita - this is what I used to build boost_1_80 

echo "Building and installing Boost libraries with c++20/libstdc++ ..."
CC="/usr/bin/clang" \
CXX="/usr/bin/clang++" \
  ./b2 include="/usr/include" library-path="/usr/lib/x86_64-linux-gnu" toolset=clang cxxflags="-std=c++20" linkflags="" --prefix=/opt/boost_1_80 --with-iostreams --with-filesystem --with-regex --with-python --with-thread --with-program_options --with-system --with-coroutine install --reconfigure
echo "Done!"

You don't have to use c++20, you can replace it with c++17 for example. But you need to ensure boost can find your ICU headers and libraries. You can verify this by analysing output. In my case I get: 

Building and installing Boost libraries with c++20/libstdc++ ...
Performing configuration checks

    - default address-model    : 64-bit [1]
    - default architecture     : x86 [1]
    - has std::atomic_ref      : yes [2]
    - has statx                : yes [2]
    - has init_priority attribute : yes [2]
    - has stat::st_blksize     : yes [2]
    - has stat::st_mtim        : yes [2]
    - has stat::st_mtimensec   : no [2]
    - has stat::st_mtimespec   : no [2]
    - has stat::st_birthtim    : no [2]
    - has stat::st_birthtimensec : no [2]
    - has stat::st_birthtimespec : no [2]
    - has fdopendir(O_NOFOLLOW) : yes [2]
    - has POSIX *at APIs       : yes [2]
    - zlib                     : yes
    - bzip2                    : yes
    - lzma                     : yes
    - zstd                     : yes
    - has_lzma_cputhreads builds : yes [2]
    - has_icu builds           : yes [2]
    - lockfree boost::atomic_flag : yes [2]
    - has std::atomic_ref      : yes [3]
    - has statx                : yes [3]
    - has init_priority attribute : yes [3]
    - has stat::st_blksize     : yes [3]
    - has stat::st_mtim        : yes [3]
    - has stat::st_mtimensec   : no [3]
    - has stat::st_mtimespec   : no [3]
    - has stat::st_birthtim    : no [3]
    - has stat::st_birthtimensec : no [3]
    - has stat::st_birthtimespec : no [3]
    - has fdopendir(O_NOFOLLOW) : yes [3]
    - has POSIX *at APIs       : yes [3]
    - zlib                     : yes [4]
    - bzip2                    : yes [4]
    - lzma                     : yes [4]
    - zstd                     : yes [4]
    - has_lzma_cputhreads builds : yes [3]
    - has_icu builds           : yes [3]
    - lockfree boost::atomic_flag : yes [3]

[1] clang-14
....

NOTE: - has_icu builds : yes [2] -- !!!

comiquita commented 1 year ago

@artemp,

There was a lot of progress today, however, I ended up having the original issue (segfault issue). Here's the summary of what I did:

I got boost with ICU support. See here:

$ ldd /usr/local/lib/libboost_regex.so
    linux-vdso.so.1 (0x00007ffe7a0d5000)
    libicudata.so.72 => /usr/local/lib/libicudata.so.72 (0x00007f802147f000)
    libicui18n.so.72 => /usr/local/lib/libicui18n.so.72 (0x00007f8021158000)
    libicuuc.so.72 => /usr/local/lib/libicuuc.so.72 (0x00007f8020f4b000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8020d05000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8020c1e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8020bfc000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f80209d4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f802329a000)

Then I configured with scons the way you did and it worked:

./scons/scons.py  CXX_STD=17
DEBUG=True 
PROJ=True
CAIRO=True
BOOST_INCLUDES=/usr/local/include/ 
BOOST_LIBS=/usr/local/lib 
XMLPARSER=libxml2
XML2_INCLUDES=/usr/local/include/libxml 
XML2_LIBS=/usr/local/lib 
SVG_RENDERER=True 
SVG2PNG=True 
INPUT_PLUGINS=all
CUSTOM_DEFINES:-DHAVE_LIBXML2 -DHAVE_CAIRO -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1 -DHAVE_JPEG -DMAPNIK_USE_PROJ -DHAVE_PNG -DHAVE_TIFF -DBOOST_REGEX_HAS_ICU

And I didn't have the Segmentation fault (core dumped) anymore. So it worked

I then did sudo ./scons/scons.py install, to install the packages and tested by writing

mapnik-config -v
4.0.0

Installed python-mapnik, and I'm back to the same issue of this thread:

$ python3 -c "import mapnik;print(mapnik.__file__)"

/home/alexander/src/python-mapnik/mapnik/__init__.py
[1]    403553 segmentation fault (core dumped)  python3 -c "import mapnik;print(mapnik.__file__)"

I'm going to try and do https://github.com/dabrahams/mpl/commit/453e0cf684097d112d54e66d2917cd4556dfeb3d

comiquita commented 1 year ago

@artemp, I tried https://github.com/dabrahams/mpl/commit/453e0cf684097d112d54e66d2917cd4556dfeb3d

And this is what I'm getting when I build boost.

In file included from ./boost/mpl/int.hpp:20:
./boost/mpl/aux_/integral_wrapper.hpp:75:23: error: use of undeclared identifier 'n'
    template<decltype(n-1) n_minus_1 = n-1>
                      ^
./boost/mpl/aux_/integral_wrapper.hpp:75:40: error: use of undeclared identifier 'n'
    template<decltype(n-1) n_minus_1 = n-1>
                                       ^
./boost/mpl/aux_/integral_wrapper.hpp:76:42: error: use of undeclared identifier 'T'
    struct prior_impl : integral_wrapper<T, static_cast<T>(n_minus_1)> {};
                                         ^
./boost/mpl/aux_/integral_wrapper.hpp:77:13: error: too few template arguments for class template 'prior_impl'
    typedef prior_impl<> prior;
            ^
./boost/mpl/aux_/integral_wrapper.hpp:76:12: note: template is declared here
    struct prior_impl : integral_wrapper<T, static_cast<T>(n_minus_1)> {};
           ^
./boost/mpl/aux_/integral_wrapper.hpp:79:23: error: use of undeclared identifier 'n'
    template<decltype(n+1) n_plus_1 = n+1>
                      ^
./boost/mpl/aux_/integral_wrapper.hpp:79:39: error: use of undeclared identifier 'n'
    template<decltype(n+1) n_plus_1 = n+1>
                                      ^
./boost/mpl/aux_/integral_wrapper.hpp:80:41: error: use of undeclared identifier 'T'
    struct next_impl : integral_wrapper<T, static_cast<T>(n_plus_1)> {};
                                        ^
./boost/mpl/aux_/integral_wrapper.hpp:81:13: error: too few template arguments for class template 'next_impl'
    typedef next_impl<> next;
            ^
./boost/mpl/aux_/integral_wrapper.hpp:80:12: note: template is declared here
    struct next_impl : integral_wrapper<T, static_cast<T>(n_plus_1)> {};
           ^

I'll revert the above and I'm going to try build everything with clang-13 (as per your comments at the beginning of this thread)

comiquita commented 1 year ago

I tried the whole process with clang-13 (building boost, mapnik and python-mapnik with clang-13), and I still get the same issue:

python3 -c "import mapnik;print(mapnik.__file__)"
/home/alexander/src/python-mapnik/mapnik/__init__.py
[1]    426348 segmentation fault (core dumped)  python3 -c "import mapnik;print(mapnik.__file__)"
comiquita commented 1 year ago

@artemp

I realized I wasn't configuring mapnik with CUSTOM_CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0

So I did: ./scons/scons.py CXX_STD="17" DEBUG=True PROJ=True CAIRO=True BOOST_INCLUDES=/usr/local/include/ BOOST_LIBS=/usr/local/lib XMLPARSER=libxml2 XML2_INCLUDES=/usr/local/include/libxml XML2_LIBS=/usr/local/lib SVG_RENDERER=True SVG2PNG=True INPUT_PLUGINS=all CUSTOM_CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 CUSTOM_DEFINES="-DHAVE_LIBXML2 -DHAVE_CAIRO -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1 -DHAVE_JPEG -DMAPNIK_USE_PROJ -DHAVE_PNG -DHAVE_TIFF -DBOOST_REGEX_HAS_ICU"

However, I had two issues:

1.
Note: will build without these OPTIONAL dependencies:
   - jpeg (JPEG C library | configure with JPEG_LIBS & JPEG_INCLUDES)

and

2.

/usr/bin/ld: test/visual/run.o:(.data.rel.ro._ZTVN5boost10wrapexceptINS_15program_options16validation_errorEEE[_ZTVN5boost10wrapexceptINS_15program_options16validation_errorEEE]+0x60): undefined reference to `boost::program_options::error_with_option_name::substitute_placeholders(std::string const&) const'
/usr/bin/ld: test/visual/run.o:(.data.rel.ro._ZTVN5boost10wrapexceptINS_15program_options20invalid_option_valueEEE[_ZTVN5boost10wrapexceptINS_15program_options20invalid_option_valueEEE]+0x60): undefined reference to `boost::program_options::error_with_option_name::substitute_placeholders(std::string const&) const'
/usr/bin/ld: warning: creating DT_TEXTREL in a PIE
collect2: error: ld returned 1 exit status
scons: *** [test/visual/run] Error 1
scons: building terminated because of errors.

Do you have any idea?

comiquita commented 1 year ago

hi @artemp , finally! I got it.

Downgraded ICU to version 70 and libproj to version 22. Then recompiled boost with that version ICU, the built webp, sqlite3, install cairo, then mapnik and python-mapik.

the config.log, helped me a lot.

I don't get any more seg faults or core dumps anymore.