C++ exceptions are not converted to python exceptions on Apple Silicon

[finsberg]

The following code is excepted to run without errors

import cppyy
import pytest

cppyy.cppdef(
    """
void throw_std_exc(){
    throw std::runtime_error("This is an exception");
}
"""
)

from cppyy.gbl import throw_std_exc

with pytest.raises(Exception) as e:
    throw_std_exc()

assert str(e.value) == 'void ::throw_std_exc() =>\n    runtime_error: This is an exception'

However, I get the following result

libc++abi: terminating with uncaught exception of type std::runtime_error: This is an exception
 *** Break *** abort
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/usr/lib/system/libsystem_pthread.dylib] pthread_kill (no debug info)
[/usr/lib/system/libsystem_c.dylib] abort (no debug info)
[/usr/lib/libc++abi.dylib] __cxxabiv1::__aligned_malloc_with_fallback(unsigned long) (no debug info)
[/usr/lib/libc++abi.dylib] demangling_terminate_handler() (no debug info)
[/usr/lib/libobjc.A.dylib] _objc_terminate() (no debug info)
[/usr/lib/libc++abi.dylib] std::__terminate(void (*)()) (no debug info)
[/usr/lib/libc++abi.dylib] __cxa_get_exception_ptr (no debug info)
[/usr/lib/libc++abi.dylib] __cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*) (no debug info)
[<unknown binary>] (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.so] WrapperCall(long, unsigned long, void*, void*, void*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::(anonymous namespace)::VoidExecutor::Execute(long, void*, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] call_function (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyEval_EvalFrameDefault (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyEval_Vector (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] PyEval_EvalCode (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyRun_SimpleFileObject (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyRun_AnyFileObject (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] Py_RunMain (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] pymain_main (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] Py_BytesMain (no debug info)
[/usr/lib/dyld] start (no debug info)
 *** Break *** abort
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/usr/lib/system/libsystem_pthread.dylib] pthread_kill (no debug info)
[/usr/lib/system/libsystem_c.dylib] abort (no debug info)
[/usr/lib/libc++abi.dylib] __cxxabiv1::__aligned_malloc_with_fallback(unsigned long) (no debug info)
[/usr/lib/libc++abi.dylib] demangling_terminate_handler() (no debug info)
[/usr/lib/libobjc.A.dylib] _objc_terminate() (no debug info)
[/usr/lib/libc++abi.dylib] std::__terminate(void (*)()) (no debug info)
[/usr/lib/libc++abi.dylib] __cxa_get_exception_ptr (no debug info)
[/usr/lib/libc++abi.dylib] __cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*) (no debug info)
[<unknown binary>] (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.so] WrapperCall(long, unsigned long, void*, void*, void*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::(anonymous namespace)::VoidExecutor::Execute(long, void*, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info)
[/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/libcppyy.cpython-310-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] call_function (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyEval_EvalFrameDefault (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyEval_Vector (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] PyEval_EvalCode (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyRun_SimpleFileObject (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] _PyRun_AnyFileObject (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] Py_RunMain (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] pymain_main (no debug info)
[/Users/finsberg/.pyenv/versions/3.10.4/bin/python3.10] Py_BytesMain (no debug info)
[/usr/lib/dyld] start (no debug info)

I am running Python version 3.10.4 on MacBook Pro M1 Max and the following version of the relevant packages

cppyy: 2.3.1
cppyy-backend: 1.14.8
cppyy-cling: 6.25.3
CPyCppyy: 1.12.10

Any ideas on how to resolve this?

[wlav]

This is known and Apple has said they won't fix it in the current JIT (see: https://github.com/llvm/llvm-project/issues/49036). I have an idea how to work around the problem, though.

[finsberg]

Thanks for a quick response. Let me know if there is anything I can do to help.

[wlav]

I've been working through several scenarios and the best I am able to do is to trap the uncaught exception and do a longjmp out of the stack from the handler. This will make examples like the one you have above work, but there is no stack unwinding of the JITed code, so it may lead to resource leaks if said JITed code expects destructors to be called.

[finsberg]

Sounds good. I think it would be good to somehow suppress this uncaught exception, but perhaps it would be good to display some warning?

In my case, this would be one of the tests in my test suite (see e.g here: https://github.com/ComputationalPhysiology/goss/blob/80c1c767a39aa76ee62f690a00af44f90a54c176/tests/test_ode_bindings.py#L54). Currently, I am just skipping this test if I am running on ARM, because otherwise it will just silently stop the test suite once it gets to this test.

[wlav]

Something like eval_component probably does not allocate anything and should be okay?

And no, I don't want to suppress the uncaught exception: it's actually rethrown after the lonjmp, so the assert in the example above succeeds. But yes, a warning about potential resource leaks is sensible (together with a way of suppressing the message).

[finsberg]

True, there is no allocation happening in that function. OK, I see, that sounds great!

[wlav]

This seems to work for me: https://github.com/wlav/cppyy-backend/commit/2fffa15de3036c40fe6946e29d11df761ca485b7

[finsberg]

Thanks for addressing this so quickly. I tried to clone the repo (cppyy-backend) and install the development version, but I keep getting this error (see below). It is probably just me that haven't set up the development environment correctly

(this is inside cppyy-backend/clingwrapper)

$ python -m pip install . -vvv                          
Using pip 22.1.2 from /Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-ephem-wheel-cache-3w_x40tv
Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx
Initialized build tracking at /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx
Created build tracker: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx
Entered build tracker: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx
Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-install-3l6hl9cw
Processing /Users/finsberg/local/src/cppyy-backend/clingwrapper
  Added file:///Users/finsberg/local/src/cppyy-backend/clingwrapper to build tracker '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx'
  Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0
  Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-standalone-pip-8i4i9hdu
  Running command pip subprocess to install build dependencies
  Using pip 22.1.2 from /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-standalone-pip-8i4i9hdu/__env_pip__.zip/pip (python 3.10)
  Collecting cppyy-cling==6.25.3
    Using cached cppyy_cling-6.25.3-py2.py3-none-macosx_11_0_arm64.whl (21.3 MB)
  Collecting setuptools
    Using cached setuptools-62.6.0-py3-none-any.whl (1.2 MB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Installing collected packages: cppyy-cling, wheel, setuptools
    Creating /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin
    changing mode of /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin/cling-config to 755
    changing mode of /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin/cppyy-generator to 755
    changing mode of /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin/genreflex to 755
    changing mode of /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin/rootcling to 755
    changing mode of /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/bin/wheel to 755
  Successfully installed cppyy-cling-6.25.3 setuptools-62.6.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  Getting requirements to build wheel ... done
  Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c
  Running command Preparing metadata (pyproject.toml)
  /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  running dist_info
  creating /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info
  writing /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/dependency_links.txt
  writing requirements to /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/requires.txt
  writing top-level names to /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/top_level.txt
  writing manifest file '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE.txt'
  writing manifest file '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend.egg-info/SOURCES.txt'
  creating '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-modern-metadata-4m44qt0c/cppyy_backend-1.14.8.dist-info'
  Preparing metadata (pyproject.toml) ... done
  Source in /Users/finsberg/local/src/cppyy-backend/clingwrapper has version 1.14.8, which satisfies requirement cppyy-backend==1.14.8 from file:///Users/finsberg/local/src/cppyy-backend/clingwrapper
  Removed cppyy-backend==1.14.8 from file:///Users/finsberg/local/src/cppyy-backend/clingwrapper from build tracker '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx'
Requirement already satisfied: cppyy-cling==6.25.3 in /Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages (from cppyy-backend==1.14.8) (6.25.3)
Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-unpack-yei4uge0
Building wheels for collected packages: cppyy-backend
  Created temporary directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-wheel-w85zjju0
  Destination directory: /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-wheel-w85zjju0
  Running command Building wheel for cppyy-backend (pyproject.toml)
  /private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  running bdist_wheel
  running build
  running build_ext
  checking for build/temp.macosx-12.2-arm64-cpython-310
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-env-ahuvweh0/overlay/lib/python3.10/site-packages/cppyy_backend/include -I/Users/finsberg/local/src/goss/venv/include -I/Users/finsberg/.pyenv/versions/3.10.4/include/python3.10 -c src/clingwrapper.cxx -o build/temp.macosx-12.2-arm64-cpython-310/src/clingwrapper.o -O2 -stdlib=libc++ -D_REENTRANT -std=c++17 -m64
  src/clingwrapper.cxx:1223:41: error: no member named 'ClassInfo_FactoryWithScope' in 'CppyyLegacy::TInterpreter'
          ClassInfo_t* ci = gInterpreter->ClassInfo_FactoryWithScope(
                            ~~~~~~~~~~~~  ^
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for cppyy-backend (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/finsberg/local/src/goss/venv/bin/python /Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/tmp7t0_ivzv
  cwd: /Users/finsberg/local/src/cppyy-backend/clingwrapper
  Building wheel for cppyy-backend (pyproject.toml) ... error
  ERROR: Failed building wheel for cppyy-backend
Failed to build cppyy-backend
ERROR: Could not build wheels for cppyy-backend, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/Users/finsberg/local/src/goss/venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 377, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for cppyy-backend, which is required to install pyproject.toml-based projects
Remote version of pip: 22.1.2
Local version of pip:  22.1.2
Was pip installed by pip? True
Removed build tracker: '/private/var/folders/l4/fpn419yn0wd534g_95tm330r0000gn/T/pip-build-tracker-h7jz3mcx'

[wlav]

For some reason it's picking up an old install of cppyy-cling (2.2.x or older) to build against. With PEP517, it should pick up cppyy-cling==6.25.3 to build against (.4, the latest, is fine as well, but the dependency currently still states .3), where that missing API was introduced. Can you upgrade cppyy-cling first:

$ python -m pip install cppyy-cling --upgrade

(pip may complain about versions, but 6.25.4 is fully backwards compatible). Then install clingwrapper. Or at that point you can also build against the local install with setuptools:

$ python setup.py install

[wlav]

Released with 2.4.0 and its dependencies.