search

bloomberg / memray

Memray is a memory profiler for Python
https://bloomberg.github.io/memray/
Apache License 2.0
13.36k stars 397 forks source link

Fail to run in macOS 19 #614

Closed peekxc closed 5 months ago

peekxc commented 5 months ago

Is there an existing issue for this?

Current Behavior

I would like to use memray, however it doesn't work at all on my system. 

# test_memray.py
import numpy as np 
x = np.zeros(100)

Running this file with the command python -m memray run test_memray.py yields an immediate seg. fault: 

Writing profile results into memray-test_memray.py.44886.bin
zsh: segmentation fault  python -m memray run test_memray.py

Though a file does get written, no recording is done; the result of memray summary memray*.bin yields: 

⠋ Calculating high watermark... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   Memray ERROR: Failed to process memory record
⠋ Processing allocation records... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--Memray ERROR: Failed to process memory record
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃                                                           ┃     <Total ┃      Total ┃            ┃ Own Memory ┃ Allocati… ┃
┃ Location                                                  ┃    Memory> ┃   Memory % ┃ Own Memory ┃          % ┃     Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
└───────────────────────────────────────────────────────────┴────────────┴────────────┴────────────┴────────────┴───────────┘

I tried the previous versions of memray as well, at least back to 1.9.0, and none of the versions seem to work

Expected Behavior

Any of the demonstrated behavior in the documentation

Steps To Reproduce

  1. Make any python file
  2. Run memray run ... on the python file
  3. Observe the seg fault

Memray Version

1.12.0

Python Version

3.11

Operating System

macOS

Anything else?

======================================== System Information ======================================== System: Darwin Node Name: MacBook-Pro-8.local Release: 19.6.0 Version: Darwin Kernel Version 19.6.0: Tue Oct 12 18:34:05 PDT 2021; root:xnu-6153.141.43~1/RELEASE_X86_64 Machine: x86_64 Processor: i386

pablogsal commented 5 months ago

I think you have something weird in your system as we run the memray test suite on Intel Macs and Apple silicon macs without any problem.

Can you try to run memray under lldb and check where is segfaulting?

pablogsal commented 5 months ago

Also, can you tell us a bit more about your system?

What macOS version are you using? What hardware are you using?

pablogsal commented 5 months ago

Processor: i386

What does this mean? Are you running a 32 bit macOS?

peekxc commented 5 months ago

Also, can you tell us a bit more about your system?

What macOS version are you using? What hardware are you using?

10.15.7 (Catalina)

What does this mean? Are you running a 32 bit macOS?

No sorry, idk why this was generated when I queried the processor. It's 64 bit macOS from 2019.

Can you try to run memray under lldb and check where is segfaulting?

This probably doesn't help, but I changed the source to: 

import numpy as np 
import memray

with memray.Tracker("output_file.bin"):
  x = np.zeros(100)

Then executed it with the following outputs:

(spri) mpiekenbrock{~/memray}% lldb python test_memray.py
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) settings set -- target.run-args  "test_memray.py"
(lldb) run
Process 49064 launched: '/Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/python' (x86_64)
Process 49064 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
Target 0: (python) stopped.
(lldb) quit

Again running the code normally just seg-faults: 

(spri) mpiekenbrock{~/laplacian_kernel/examples}% python test_memray.py
zsh: segmentation fault  python test_memray.py

I'm not entirely sure how to track down the segfault

pablogsal commented 5 months ago

When you see this in lldb:

error: memory read failed for 0x0

type “bt” and paste the output here

peekxc commented 5 months ago

type “bt” and paste the output here

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x0000000116615fb9 _memray.cpython-311-darwin.so`memray::linker::patch_symbols_in_shared_object(mach_header const*, long, char const*, bool, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&) + 3881
    frame #2: 0x0000000116614e16 _memray.cpython-311-darwin.so`memray::linker::patch_symbols_in_all_shared_objects(bool, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&) + 726
    frame #3: 0x0000000116645811 _memray.cpython-311-darwin.so`memray::tracking_api::Tracker::Tracker(std::__1::unique_ptr<memray::tracking_api::RecordWriter, std::__1::default_delete<memray::tracking_api::RecordWriter> >, bool, unsigned int, bool, bool) + 3665
    frame #4: 0x00000001165f4c28 _memray.cpython-311-darwin.so`__pyx_pw_6memray_7_memray_7Tracker_3__enter__(_object*, _object* const*, long, _object*) + 1000
    frame #5: 0x0000000100078ef1 python`method_vectorcall + 449
    frame #6: 0x00000001001c6442 python`_PyEval_EvalFrameDefault + 217906
    frame #7: 0x000000010018ea5b python`_PyEval_Vector + 1275
    frame #8: 0x000000010018e4ea python`PyEval_EvalCode + 250
    frame #9: 0x000000010023d787 python`run_mod + 167
    frame #10: 0x000000010023d595 python`pyrun_file + 133
    frame #11: 0x000000010023d073 python`_PyRun_SimpleFileObject + 275
    frame #12: 0x000000010023ca1f python`_PyRun_AnyFileObject + 143
    frame #13: 0x0000000100261a6b python`pymain_run_file_obj + 267
    frame #14: 0x0000000100261485 python`pymain_run_file + 85
    frame #15: 0x0000000100260dfd python`Py_RunMain + 1965
    frame #16: 0x00000001000018a8 python`main + 56
    frame #17: 0x00007fff723becc9 libdyld.dylib`start + 1
    frame #18: 0x00007fff723becc9 libdyld.dylib`start + 1
pablogsal commented 5 months ago

Oh, interesting, seems that it's segfaulting at some point in patch_symbols_in_shared_object. I have a guess but I need you to build memray in debug mode and re-run lldb. Can you try to follow these build steps:

$ brew install lz4
$ export MEMRAY_MINIMIZE_INLINING=1
$ export CFLAGS="-O0 -g3 -I$(brew --prefix lz4)/include" LDFLAGS="-L$(brew --prefix lz4)/lib -Wl,-rpath,$(brew --prefix lz4)/lib"
$ git clone git@github.com:bloomberg/memray.git 
$ python3 -m venv venv
$ source venv/bin/activate
$ cd memray
$ pip install -e .

And then re-run lldb over your example, you shold have source now so we can check where is segfaulting

peekxc commented 5 months ago

Ahh it looks like I might need to update to OS X 10.14.

I failed to compile memray from source using gcc-12 and clang-15. The relevant compiler error(s) in question: 

 In file included from src/memray/_memray.cpp:1361:
    src/memray/_memray/snapshot.h:110:59: error: 'value' is unavailable: introduced in macOS 10.14
                const auto& intersection = maybe_intersection.value();
                                                              ^
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/optional:953:27: note: 'value' has been explicitly marked unavailable here
        constexpr value_type& value() &
                              ^
    In file included from src/memray/_memray.cpp:1377:
    src/memray/_memray/tracking_api.h:222:19: error: 'value' is unavailable: introduced in macOS 10.14
                trace.value().fill(1);
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/optional:953:27: note: 'value' has been explicitly marked unavailable here
        constexpr value_type& value() &
                              ^
...

For full reference, here's my compiler config: 

(venv) mpiekenbrock{~/memray}% g++ --version
g++-12 (Homebrew GCC 12.2.0) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(venv) mpiekenbrock{~/memray}% clang++ --version
Homebrew clang version 15.0.7
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin
pablogsal commented 5 months ago

I failed to compile memray from source using gcc-12 and clang-15. The relevant compiler error(s) in question:

Looks like your XCODE installation is broken somehow (which you can confirm by looking up those error messages)

pablogsal commented 5 months ago

Try setting $ export MACOSX_DEPLOYMENT_TARGET=10.14

pablogsal commented 5 months ago

You need to figure out how to fix that compilation problem because we cannot reproduce it on our side nor in our CI, so getting it to build on your system it's our only option at debugging this.

pablogsal commented 5 months ago

Ahh it looks like I might need to update to OS X 10.14.

But you said you are using 10.15.7 (Catalina) no? I think you just need to export MACOSX_DEPLOYMENT_TARGET as I mentioned before

peekxc commented 5 months ago

Try setting $ export MACOSX_DEPLOYMENT_TARGET=10.14

I did follow this originally, actually, though I thought I was meant to replace 10.14 with my OS's version; exporting thing does get past that error.

You need to figure out how to fix that compilation problem because we cannot reproduce it on our side nor in our CI, so getting it to build on your system it's our only option at debugging this.

Yeah, that's fair. I might end up updating my OS to macOS 14 just to see if that fixes it. The newest error after fixing the one above is this: 

    src/memray/_memray/macho_shenanigans.cpp:299:17: error: use of undeclared identifier '_dyld_shared_cache_contains_path'
                if (_dyld_shared_cache_contains_path(image_name) && strcmp(section->segname, SEG_TEXT) == 0)
                    ^
1 warning and 1 error generated.

Which I guess may be related to this; will check back in once I try to fix the source

pablogsal commented 5 months ago

Oh, that more or less confirms my guess: that function is not available at runtime in your system.

As a quick test can you substitute _dyld_shared_cache_contains_path(image_name) with 0:

                if (0 && strcmp(section->segname, SEG_TEXT) == 0)

and check if it works?

peekxc commented 5 months ago

It works :), everything seems to be fine now, have run a couple benchmarks to be sure.

Closing this issue. Thank you for your expediency

pablogsal commented 5 months ago

Leave the issue open for now as we still need to fix it on our side

pablogsal commented 5 months ago

It works :), everything seems to be fine now, have run a couple benchmarks to be sure.

Closing this issue. Thank you for your expediency

Awesome! If you want to use your version I suggest that you recompile memray without the debug options so you don’t run a slow version

pablogsal commented 5 months ago

Fixed by #615