Compiling with -fvisibilty=hidden on macOS breaks hashing

[saraedum]

When building a shared library on macOS with ~-g~ -fvisibility=hidden and the conda-forge compilers, the symbol table contains some unresolved symbols. I guess this is the expected behaviour somehow but I don't understand how macOS works here:

000000000002e1a0 lw    F __TEXT,__text std::__1::hash<exactreal::Element<exactreal::RationalField> >::operator()(exactreal::Element<exactreal::RationalField> const&) const
000000000002e1a0 l    d  *UND* std::__1::hash<exactreal::Element<exactreal::RationalField> >::operator()(exactreal::Element<exactreal::RationalField> const&) const

Invoking this hash function from cppyy prints a warning and produces the wrong result:

>>> from pyexactreal import ExactReals
>>> x = ExactReals().random_element()
>>> import cppyy
>>> cppyy.gbl.std.hash[type(x._backend)]()(x._backend)
IncrementalExecutor::executeFunction: symbol '_ZNKSt3__14hashIN9exactreal7ElementINS1_13RationalFieldEEEEclERKS4_' unresolved while linking symbol '__cf_27'!
You are probably missing the definition of std::__1::hash<exactreal::Element<exactreal::RationalField> >::operator()(exactreal::Element<exactreal::RationalField> const&) const
Maybe you need to load the corresponding shared library?
18446744073709551615

When I invoke this method indirectly through Python's hash(), I get:

>>> hash(x._backend)
SystemError: <built-in function hash> returned NULL without setting an error

As soon as I drop -fvisibility=hidden from my CXXFLAGS when building the shared library, everything works.

I originally reported that this was caused by -g. I got myself confused by looking at too many CI runs here. I can reproduce this locally with -fvisibility=hidden in macOS.

[saraedum]

If there are no immediate ideas what might be going on here. I could try to borrow a Mac and create a proper reproducer for this one.

[wlav]

If the shared library does not contain the symbols in the first place, then I'm not sure how they could show up in the JIT later? Are these functions inline?

That return value of 18446744073709551615 is 2**64-1. Smells like a -1 return (error) through an unsigned type. Still, Py_hash_t is Py_ssize_t, which is signed?

Not sure about how the 2nd error came about: the tp_hash implementation returns a Py_hash_t value, not a PyObject* and can thus not return a NULL result?

What if you add -g to EXTRA_CLING_ARGS to match the debug build? (By default, if not set, it will add -O2 and -mavx if applicable.

$ export EXTRA_CLING_ARGS="-O2 -g -mavx"

[saraedum]

What if you add -g to EXTRA_CLING_ARGS to match the debug build? (By default, if not set, it will add -O2 and -mavx if applicable.

$ export EXTRA_CLING_ARGS="-O2 -g -mavx"

That does not make any difference.

I also tried with the cppyy from pypi. The issue is the same. Additionally, I get a segfault in one test:

ERROR in cling::CIFactory::createCI(): cannot extract standard library include paths!
Invoking:
  LC_ALL=C /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++    -xc++ -E -v /dev/null 2>&1 | sed -n -e '/^.*include/,${' -e '/^ \/.*++/p' -e '}'
Results was:
With exit code 0
Warning: sysroot "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" not found (ignoring for now).input_line_1:1:10: fatal error: 'new' file not found
#include 
         ^~~~~
Warning in cling::IncrementalParser::CheckABICompatibility():
  Failed to extract C++ standard library version.
In file included from input_line_3:1:
In file included from ./include/RtypesCore.h:23:
/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/etc//cling/lib/clang/9.0.1/include/stddef.h:35:9: error: unknown type name '__PTRDIFF_TYPE__'
typedef __PTRDIFF_TYPE__ ptrdiff_t;
        ^
/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/etc//cling/lib/clang/9.0.1/include/stddef.h:46:9: error: unknown type name '__SIZE_TYPE__'
typedef __SIZE_TYPE__ size_t;
        ^
/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/etc//cling/lib/clang/9.0.1/include/stddef.h:74:9: error: unknown type name '__WCHAR_TYPE__'
typedef __WCHAR_TYPE__ wchar_t;
        ^
input_line_3:2:10: fatal error: 'string' file not found
#include 
         ^~~~~~~~
0  libCling.so              0x00000001128fbe45 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 37
1  libCling.so              0x00000001128fb0c7 llvm::sys::RunSignalHandlers() + 39
2  libCling.so              0x00000001128fc3e2 SignalHandler(int) + 258
3  libsystem_platform.dylib 0x00007fff69d7b5fd _sigtramp + 29
4  libsystem_platform.dylib 0x00007f86a25069e8 _sigtramp + 18446743554965943304
5  libCling.so              0x0000000110310d68 CppyyLegacy::TMetaUtils::TNormalizedCtxt::TNormalizedCtxt(cling::LookupHelper const&) + 40
6  libCling.so              0x00000001103ca4ef CppyyLegacy::TCling::TCling(char const*, char const*, char const* const*) + 12655
7  libCling.so              0x00000001103c30d5 CreateInterpreter + 53
8  libCoreLegacy.so         0x00000001139f0a26 CppyyLegacy::TROOT::InitInterpreter() + 214
9  libCoreLegacy.so         0x00000001139f0902 CppyyLegacy::Internal::GetROOT2() + 34
10 libCling.so              0x00000001103a707d TCling__GetInterpreter + 61
11 libCling.so              0x000000011036fb77 RootClingMain(int, char**, bool) + 18887
12 libCling.so              0x000000011037fd24 ROOT_rootcling_Driver + 340
13 rootcling                0x000000010ff6ff5e main + 142
14 libdyld.dylib            0x00007fff69b7ecc9 start + 1
Stack dump:
0.  Program arguments: ./bin/rootcling -rootbuild -generate-pch -f /var/folders/6l/yzyyrq255159psqqtdfk00pm0000gn/T/allDict.cxx -noDictSelection -D__CLING__ -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DROOT_PCH -I./include -I./etc -I./etc/dictpch -I./etc/cling -I/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/include -cxxflags -m64 -pipe -fsigned-char -fno-common -Qunused-arguments -pthread etc/dictpch/allHeaders.h etc/dictpch/allLinkDefs.h
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[] (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] CppyyLegacy::TMetaUtils::TNormalizedCtxt::TNormalizedCtxt(cling::LookupHelper const&) (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] CppyyLegacy::TCling::TCling(char const*, char const*, char const* const*) (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] CreateInterpreter (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TROOT::InitInterpreter() (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::Internal::GetROOT2() (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] TCling__GetInterpreter (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] RootClingMain(int, char**, bool) (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so] ROOT_rootcling_Driver (no debug info)
[/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/./bin/rootcling] main (no debug info)
[/usr/lib/system/libdyld.dylib] start (no debug info)
/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/loader.py:132: UserWarning: Precompiled header may be out of date (failed to build).
  warnings.warn('Precompiled header may be out of date (%s).' % msg)
Warning: sysroot "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" not found (ignoring for now).error: Problems loading PCH: '/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy_backend/etc/allDict.cxx.pch.6.25.1'.
(Re-)building pre-compiled headers (options:-I/Users/hackintosh/proj/exact-real/pyexactreal/test/../../libexactreal -I/Users/hackintosh/proj/exact-real/pyexactreal/test/../../libexactreal/exact-real  -UNDEBUG -g3 -fvisibility=hidden -fvisibility-inlines-hidden -std=c++17 -UNDEBUG -g3 -fvisibility=hidden -fvisibility-inlines-hidden); this may take a minute ...
============================= test session starts ==============================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/hackintosh/proj/exact-real/pyexactreal/test
collected 3 items

module.py Fatal Python error: Segmentation fault

Current thread 0x00000001154c6dc0 (most recent call first):
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/cppyy/_cpython_cppyy.py", line 79 in __getitem__
  File "/Users/hackintosh/proj/exact-real/pyexactreal/src/pyexactreal/cppyy_exactreal.py", line 403 in makeModule
  File "/Users/hackintosh/proj/exact-real/pyexactreal/src/pyexactreal/cppyy_exactreal.py", line 421 in ZZModule
  File "/Users/hackintosh/proj/exact-real/pyexactreal/test/module.py", line 31 in test_ZZ_module
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/python.py", line 1641 in runtest
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 255 in 
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 311 in from_call
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/main.py", line 323 in _main
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/hackintosh/micromamba/envs/test/envs/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/Users/hackintosh/proj/exact-real/pyexactreal/test/./module.py", line 61 in 
FAIL module.py (exit status: 139)

[saraedum]

If the shared library does not contain the symbols in the first place, then I'm not sure how they could show up in the JIT later? Are these functions inline?

No. They are not inline. They are explicitly instantiated in a .cpp file when building the shared library.

[saraedum]

@wlav sorry for reporting that this was caused by -g originally. The culprit is -fvisibility=hidden. The methods in question are marked as visible and that works on Linux. I'll double check that these methods can actually be linked against from C++ without using cppyy on macOS.

[saraedum]

So it turns out that visibility of symbols on macOS/clang works differently than it does on Linux.

The hashing function is simply not exported in the .dylib and that's why cppyy cannot find it. So there is nothing wrong with cppyy here.

I am sorry for the noise.