Importing cppyy before xgboost breaks xgboost

[guitargeek]

Reproducer:

• Create new virtual environment
• Install packages with pip install cppyy xgboost numpy scikit-learn
• Run this code:

import numpy as np
import cppyy
from xgboost import XGBClassifier

n = 10000

x = np.random.random((n, 4))
y = np.random.randint(2, size=n)

XGBClassifier(max_depth=3, n_estimators=10).fit(x, y)

Swapping the imports fixes the crash, and it also doesn't happen with xgboost before version 2.0.0.

Happens for me on Arch Linux with Python 3.11.8.

Note that it's not necessarily on cppyy to fix this, but I just wanted to document it somewhere.

Stack trace extract:

free(): invalid pointer
 *** Break *** abort

...

#8  0x0000723e0b9f832c in ?? () from /usr/lib/libc.so.6
#9  0x0000723e0b9a76c8 in raise () from /usr/lib/libc.so.6
#10 0x0000723e0b98f4b8 in abort () from /usr/lib/libc.so.6
#11 0x0000723e0b990395 in ?? () from /usr/lib/libc.so.6
#12 0x0000723e0ba022a7 in ?? () from /usr/lib/libc.so.6
#13 0x0000723e0ba045b4 in ?? () from /usr/lib/libc.so.6
#14 0x0000723e0ba06e93 in free () from /usr/lib/libc.so.6
#15 0x0000723d5eaec241 in std::__detail::_Compiler<std::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type) () from /home/rembserj/.virtualenvs/repro/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#16 0x0000723d5eabd99d in xgboost::Context::SetDeviceOrdinal(std::vector<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) () from /home/rembserj/.virtualenvs/repro/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#17 0x0000723d5ebf8601 in xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int) () from /home/rembserj/.virtualenvs/repro/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#18 0x0000723d5eba2a07 in xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int) () from /home/rembserj/.virtualenvs/repro/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#19 0x0000723d5e965c40 in XGQuantileDMatrixCreateFromCallback () from /home/rembserj/.virtualenvs/repro/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#20 0x0000723e0acdc596 in ?? () from /usr/lib/libffi.so.8
#21 0x0000723e0acd900e in ?? () from /usr/lib/libffi.so.8
#22 0x0000723e0acdbbd3 in ffi_call () from /usr/lib/libffi.so.8
#23 0x0000723e0ace6011 in ?? () from /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#24 0x0000723e0ace5833 in ?? () from /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#25 0x0000723e0b55b0dc in _PyObject_MakeTpCall () from /usr/lib/libpython3.11.so.1.0
#26 0x0000723e0b5665d3 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#27 0x0000723e0b58d9b0 in _PyFunction_Vectorcall () from /usr/lib/libpython3.11.so.1.0
#28 0x0000723e0b5983dc in PyObject_Call () from /usr/lib/libpython3.11.so.1.0
#29 0x0000723e0b56a64d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#30 0x0000723e0b58d9b0 in _PyFunction_Vectorcall () from /usr/lib/libpython3.11.so.1.0
#31 0x0000723e0b595275 in ?? () from /usr/lib/libpython3.11.so.1.0
#32 0x0000723e0b55b3da in ?? () from /usr/lib/libpython3.11.so.1.0
#33 0x0000723e0b598366 in PyObject_Call () from /usr/lib/libpython3.11.so.1.0
#34 0x0000723e0b56a64d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#35 0x0000723e0b58d9b0 in _PyFunction_Vectorcall () from /usr/lib/libpython3.11.so.1.0
#36 0x0000723e0b5983dc in PyObject_Call () from /usr/lib/libpython3.11.so.1.0
#37 0x0000723e0b56a64d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#38 0x0000723e0b58d9b0 in _PyFunction_Vectorcall () from /usr/lib/libpython3.11.so.1.0
#39 0x0000723e0b5983dc in PyObject_Call () from /usr/lib/libpython3.11.so.1.0
#40 0x0000723e0b56a64d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#41 0x0000723e0b61fae4 in ?? () from /usr/lib/libpython3.11.so.1.0
#42 0x0000723e0b61f4cc in PyEval_EvalCode () from /usr/lib/libpython3.11.so.1.0
#43 0x0000723e0b63cd03 in ?? () from /usr/lib/libpython3.11.so.1.0
#44 0x0000723e0b638e0a in ?? () from /usr/lib/libpython3.11.so.1.0
#45 0x0000723e0b64f383 in ?? () from /usr/lib/libpython3.11.so.1.0
#46 0x0000723e0b64ecf5 in _PyRun_SimpleFileObject () from /usr/lib/libpython3.11.so.1.0
#47 0x0000723e0b64d5f8 in _PyRun_AnyFileObject () from /usr/lib/libpython3.11.so.1.0
#48 0x0000723e0b648098 in Py_RunMain () from /usr/lib/libpython3.11.so.1.0
#49 0x0000723e0b6131eb in Py_BytesMain () from /usr/lib/libpython3.11.so.1.0
#50 0x0000723e0b990cd0 in ?? () from /usr/lib/libc.so.6
#51 0x0000723e0b990d8a in __libc_start_main () from /usr/lib/libc.so.6
#52 0x00005ac156671045 in _start ()

[wlav]

Just adding for the record what we already discussed: it's the use of C++ standard regex in Converters.cxx and regex symbols being exported. It's probably doable to reduce the symbols that libcppyy exports (currently simply all, but what's needed is only the stuff provided in API.h). Removing the use of regex in Converters.cxx would be unfortunate, but not the end of the world.

No idea why the use of a standard library feature would clash in this way, esp. since the use in libcppyy is built locally, so is guaranteed to use the system-local libstdc++.