A crash in `memray flamegraph` (Python 3.12.0, macOS, native mode)

neutrinoceros commented 7 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Trying out running with --native on macOS + Python 3.12, memray flamegraph chocked on the report.

Writing profile results into report.bin
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

/Users/clm/.pyenv/versions/tmp/bin/python -m memray flamegraph report.bin
⚠  No symbol information was found for the Python interpreter  ⚠

Without symbolic information reports showing native traces may not accurately reflect stack traces. Please use an
interpreter built with debug symbols for best results. Check https://bloomberg.github.io/memray/native_mode.html for more
information regarding how memray resolves symbols.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/__main__.py", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/commands/__init__.py", line 138, in main
    arg_values.entrypoint(arg_values, parser)
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/commands/common.py", line 282, in run
    self.write_report(
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/commands/common.py", line 181, in write_report
    reporter = self.reporter_factory(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/reporters/flamegraph.py", line 358, in from_snapshot
    return cls._from_any_snapshot(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/tmp/lib/python3.12/site-packages/memray/reporters/flamegraph.py", line 287, in _from_any_snapshot
    tuple(record.hybrid_stack_trace())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/memray/_memray.pyx", line 339, in memray._memray.AllocationRecord.hybrid_stack_trace
  File "src/memray/_memray.pyx", line 246, in memray._memray.hybrid_stack_trace
AssertionError

Expected Behavior

I would expect a flamegraph could still be generated (albeit with a lot of restrictions due to running on macOS with a non-debug interpreter...)

Steps To Reproduce

Environment

running on macOS 14.3.1 ARM (M2)
Python 3.12.0 installed via pyenv (pyenv installed via homebrew) -> no debugging options whatsoever
with astropy 6.0.0 + numpy 1.26.4 + memray 1.11.0 installed from PyPI (via pip)

Repr

I was running the following script, which I include verbatim for now, but I'm willing to bet that a minimal reproducer would be a lot simpler

# t.py
from astropy.table.table_helpers import simple_table
import numpy as np

TABLE_SIZE = 250_000
NCOLS = 26
LOOP_SIZE = 10
SAMPLE_SIZE = 240_000
t = simple_table(size=TABLE_SIZE, cols=NCOLS)
t.add_index("a")
for _ in range(LOOP_SIZE):
    t2 = t[np.random.randint(SAMPLE_SIZE, size=TABLE_SIZE)]

Memray Version

1.11.0

Python Version

3.12

Operating System

macOS

Anything else?

No response

neutrinoceros commented 7 months ago

Actually while writing this, I also tried with Python 3.12.1 and the problem went away, so maybe this is actually a symptom of an already fixed CPython bug ? In any case, feel free to close this, I just thought this could be valuable, albeit very niche, feedback :)

neutrinoceros commented 7 months ago

also works with Python 3.12.2 (better safe than sorry)

pablogsal commented 7 months ago

Thanks for the report and the follow ups! I will investigate nevertheless in case it’s a transient failure

pablogsal commented 6 months ago

Played a bit and couldn't reproduce, so I am closing this. Let us know if this happens again

bloomberg / memray