Python bindings on MacOS segfaults

naufik commented 1 month ago

Hi all, I'm having issue following the guide for Python Bindings and MacOS in the manual file. I'll preface this by saying that I am unfamiliar with the cmake build system - but I tried best to trouble shoot.

Steps followed

# Taken from PDF manual
# See below for example steps to build the Python bindings using CMake:
mkdir build
cd build
cmake - DTRACY_STATIC=OFF - DTRACY_CLIENT_PYTHON=ON ../

# Once this has finished building the Python package can be built as follows:
cd ../ python
python3 setup .py bdist_wheel

export PYTHONPATH=tracy/python

# test.py file
import tracy_client as tracy
sleep(50)

Observations

It seems that Cmake does not build the .so files, attempting to import the library after building only using cmake gives the following error:
```
ModuleNotFoundError: No module named tracy_client.TracyClientBindings
```

Makes sense, those are not there yet at the moment.

Ran make on the build folder. Got the following errors:


/tracy/python/bindings/NameBuffer.hpp:16:29: error: no template named 'optional' in namespace 'std'
using OptionalNumber = std::optional<std::size_t>;
                   ~~~~~^
/tracy/python/bindings/NameBuffer.hpp:17:31: error: use of undeclared identifier 'OptionalNumber'
using BufferEntry = std::pair<OptionalNumber, const char*>;
                          ^
/tracy/python/bindings/NameBuffer.hpp:21:17: error: unknown type name 'BufferEntry'
static inline BufferEntry Add(const std::string& name) {
            ^

[... redacted ... - all the errors are similar]

fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. make[2]: [python/CMakeFiles/TracyClientBindings.dir/bindings/Module.cpp.o] Error 1 make[1]: [python/CMakeFiles/TracyClientBindings.dir/all] Error 2


- Some googling says this is because `Optional` et al are only supported on C++17 and up. So I added the following line to `python/CMakeLists.txt` (chose 20 for parity reasons as all the cmakefiles seem to be set on 20)

set(CMAKE_CXX_STANDARD 20)


- ran `make` again and it seems to have successfully build. Creating the following files:

TracyClientBindings.cpython-312-darwin.so libTracyClient.0.11.0.dylib libTracyClient.dylib


- Load up python interpreter and ran `test.py` file above

segmentation fault



- I attempted to instead build the wheel and use `pip install` to load the package into my environment -- but looks like the segmentation fault still happens

## System
MacOS Sonoma 14.5
Python 3.12.2 (Installed With Homebrew)

Let me know if you have any questions or pointers. I am inexperienced in C++ debugging for segfaults but I may give it a shot once I have more time.

Thanks for building such cool stuff

wolfpld commented 1 month ago

@Chekov2k Can you take a look?

Chekov2k commented 1 month ago

Hi, I can reproduce the issue with the CXX standard if the default system compiler is used (I had used brew llvm and that was hiding the issue). I have raised an MR to fix that. Could you confirm which python version you are using? Mine is linking against home-brew python: Found Python: /opt/homebrew/Frameworks/Python.framework/Versions/3.12/bin/python3.12 (found suitable version "3.12.4", minimum required is "3.6")

naufik commented 1 month ago

Hi, @Chekov2k thanks for looking at this.

The issue happened on Python 3.12.2 and the binding links against it (also homebrew installed)

Found Python: /opt/homebrew/Frameworks/Python.framework/Versions/3.12/bin/python3.12 (found suitable version "3.12.2", minimum required is "3.6")

Chekov2k commented 1 month ago

hm, I have attached my full build log below. The resulting test.py is not segfaulting. Any chance you can compare the output and maybe send a backtrace from lldb?

build % git status
HEAD detached at v0.11.0
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   ../python/CMakeLists.txt

no changes added to commit (use "git add" and/or "git commit -a")
build % git diff
diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index e1ca2a74..7d50a32c 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -1,6 +1,9 @@
+set(CMAKE_CXX_STANDARD 20)
+
 option(EXTERNAL_PYBIND11 "Whether to download pybind11" ON)

 if(EXTERNAL_PYBIND11)
+    set(Python_FIND_FRAMEWORK "LAST")
     find_package(Python 3.6 COMPONENTS Interpreter Development REQUIRED)

     include(FetchContent)
build % cmake -DTRACY_STATIC=OFF -DTRACY_CLIENT_PYTHON=ON ../
Parsing public/common/TracyVersion.hpp file
VERSION 0.11.0
-- TRACY_ENABLE: ON
-- TRACY_ON_DEMAND: OFF
-- TRACY_CALLSTACK: OFF
-- TRACY_NO_CALLSTACK: OFF
-- TRACY_NO_CALLSTACK_INLINES: OFF
-- TRACY_ONLY_LOCALHOST: OFF
-- TRACY_NO_BROADCAST: OFF
-- TRACY_ONLY_IPV4: OFF
-- TRACY_NO_CODE_TRANSFER: OFF
-- TRACY_NO_CONTEXT_SWITCH: OFF
-- TRACY_NO_EXIT: OFF
-- TRACY_NO_SAMPLING: OFF
-- TRACY_NO_VERIFY: OFF
-- TRACY_NO_VSYNC_CAPTURE: OFF
-- TRACY_NO_FRAME_IMAGE: OFF
-- TRACY_NO_SYSTEM_TRACING: OFF
-- TRACY_PATCHABLE_NOPSLEDS: OFF
-- TRACY_DELAYED_INIT: OFF
-- TRACY_MANUAL_LIFETIME: OFF
-- TRACY_FIBERS: OFF
-- TRACY_NO_CRASH_HANDLER: OFF
-- TRACY_TIMER_FALLBACK: OFF
-- TRACY_LIBUNWIND_BACKTRACE: OFF
-- TRACY_SYMBOL_OFFLINE_RESOLVE: OFF
-- TRACY_LIBBACKTRACE_ELF_DYNLOAD_SUPPORT: OFF
-- TRACY_DEMANGLE: OFF
-- pybind11 v2.11.1
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: tracy/build
build % make
[ 25%] Building CXX object CMakeFiles/TracyClient.dir/public/TracyClient.cpp.o
In file included from tracy/public/TracyClient.cpp:14:
tracy/public/common/TracySystem.cpp:285:3: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
  sprintf( buf, "%" PRIu32, id );
  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:539:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "OS: Darwin (OSX)\n" );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:564:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "Compiler: clang %i.%i.%i\n", __clang_major__, __clang_minor__, __clang_patchlevel__ );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:605:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "User: %s@%s\n", user, hostname );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:613:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "Arch: ARM64\n" );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:690:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "CPU: unknown\n" );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:700:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "CPU cores: %i\n", std::thread::hardware_concurrency() );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:23:
tracy/public/client/TracyProfiler.cpp:719:12: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    ptr += sprintf( ptr, "RAM: %zu MB\n", memSize / 1024 / 1024 );
           ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:24:
tracy/public/client/TracyCallstack.cpp:1172:33: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
            const auto offlen = sprintf( buf, " + %td", symoff );
                                ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:28:
tracy/public/common/TracySocket.cpp:167:5: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    sprintf( portbuf, "%" PRIu16, port );
    ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:28:
tracy/public/common/TracySocket.cpp:239:5: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    sprintf( portbuf, "%" PRIu16, port );
    ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:28:
tracy/public/common/TracySocket.cpp:468:5: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    sprintf( portbuf, "%" PRIu16, port );
    ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
In file included from tracy/public/TracyClient.cpp:28:
tracy/public/common/TracySocket.cpp:578:5: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    sprintf( portbuf, "%" PRIu16, port );
    ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/stdio.h:180:1: note: 'sprintf' has been explicitly marked deprecated here
__deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
        #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                      ^
13 warnings generated.
[ 50%] Linking CXX shared library libTracyClient.dylib
[ 50%] Built target TracyClient
[ 75%] Building CXX object python/CMakeFiles/TracyClientBindings.dir/bindings/Module.cpp.o
[100%] Linking CXX shared library TracyClientBindings.cpython-312-darwin.so
[100%] Built target TracyClientBindings
build % echo $PYTHONPATH
../python
build % python3 ~/Downloads/test.py
^CTraceback (most recent call last):
  File "Downloads/test.py", line 74, in <module>
    main()
  File "Downloads/test.py", line 68, in main
    work()
  File "tracy/python/tracy_client/scoped.py", line 112, in wrapped
    value = function(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tracy/python/tracy_client/scoped.py", line 79, in wrapped
    value = function(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Downloads/test.py", line 14, in work
    sleep(0.05)
KeyboardInterrupt

naufik commented 1 month ago

I did some more testing, unfortunately I have not had time yet to dive deep with a debugger.

However the segfault no longer appears when Python is upgraded to v3.12.4 from v3.12.2... I still need to get it to build for 3.12.2 though, I'll let you know how that goes as this might be an environment hygiene issue

Also thanks so much for updating the cmake issue quickly!

Chekov2k commented 1 month ago

Could you try updating the pybind11 version to 2.13.1 (if you haven't got it installed on the OS directly) and see if that helps with the segfault? It's defined in the python CMakeFile

wolfpld / tracy

Python bindings on MacOS segfaults #829

Steps followed

Observations