bloomberg / memray

Memray is a memory profiler for Python
https://bloomberg.github.io/memray/
Apache License 2.0
13.17k stars 392 forks source link

Official Debian package #518

Closed alexmyczko closed 8 months ago

alexmyczko commented 8 months ago

Is there an existing issue for this?

Current Behavior

I am able to build 1.10.0 but get these lintian errors and warnings:

E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_inject.abi3.so]
E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_memray.cpython-311-x86_64-linux-gnu.so]
E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_memray.cpython-312-x86_64-linux-gnu.so]
E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_test_utils.cpython-311-x86_64-linux-gnu.so]
E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_test_utils.cpython-312-x86_64-linux-gnu.so]
E: memray: helper-templates-in-copyright
E: memray: missing-dependency-on-libc needed by usr/lib/python3/dist-packages/memray/_inject.abi3.so and 4 others
E: memray: privacy-breach-uses-embedded-file You may use the libjs-bootstrap package. (https://cdn.jsdelivr.net/npm/bootstrap@4.6.0/dist/css/bootstrap.min.css)
E: memray: privacy-breach-uses-embedded-file You may use the libjs-jquery package. (https://cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js) [usr/lib/pyth
E: memray source: source-is-missing [docs/_static/flamegraphs/memray-flamegraph-fib.html]
E: memray source: source-is-missing [docs/_static/flamegraphs/memray-flamegraph-mandelbrot.html]
E: memray source: source-is-missing [docs/_static/flamegraphs/memray-flamegraph-nbody.html]
E: memray source: source-is-missing [docs/_static/flamegraphs/memray-flamegraph-sqlite.html]
E: memray source: source-is-missing [src/memray/reporters/templates/assets/flamegraph.js]
E: memray source: source-is-missing [src/memray/reporters/templates/assets/flamegraph_common.js]
E: memray source: source-is-missing [src/memray/reporters/templates/assets/table.js]
E: memray source: source-is-missing [src/memray/reporters/templates/assets/temporal_flamegraph.js]
E: memray: unstripped-binary-or-object [usr/lib/python3/dist-packages/memray/_inject.abi3.so]
E: memray: unstripped-binary-or-object [usr/lib/python3/dist-packages/memray/_memray.cpython-311-x86_64-linux-gnu.so]
E: memray: unstripped-binary-or-object [usr/lib/python3/dist-packages/memray/_memray.cpython-312-x86_64-linux-gnu.so]
E: memray: unstripped-binary-or-object [usr/lib/python3/dist-packages/memray/_test_utils.cpython-311-x86_64-linux-gnu.so]
E: memray: unstripped-binary-or-object [usr/lib/python3/dist-packages/memray/_test_utils.cpython-312-x86_64-linux-gnu.so]
W: memray source: dh-make-template-in-source [debian/manpage.1.ex]
W: memray: no-manual-page [usr/bin/memray3.11]
W: memray: no-manual-page [usr/bin/memray3.12]
W: memray: no-manual-page [usr/bin/memray]
W: memray: privacy-breach-generic [<script src="https://cdn.datatables.net/1.10.23/js/datatables.bootstrap4.min.js">] (https://cdn.datatables.net/1.10.23/js/da
W: memray: privacy-breach-generic [<script src="https://cdn.datatables.net/1.10.23/js/jquery.datatables.min.js">] (https://cdn.datatables.net/1.10.23/js/jquery
W: memray: privacy-breach-generic [<script src="https://cdn.jsdelivr.net/npm/d3-flame-graph@4.0.6/dist/d3-flamegraph.min.js">] (https://cdn.jsdelivr.net/npm/d3
W: memray: privacy-breach-generic [<script src="https://cdn.jsdelivr.net/npm/d3-tip@0.9.1/dist/index.min.js">] (https://cdn.jsdelivr.net/npm/d3-tip@0.9.1/dist/
W: memray: privacy-breach-generic [<script src="https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js">] (https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodas
W: memray: privacy-breach-generic [<script src="https://cdn.jsdelivr.net/npm/plotly.js@2.11.1/dist/plotly.min.js">] (https://cdn.jsdelivr.net/npm/plotly.js@2.1
W: memray: privacy-breach-generic [<script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.1/dist/umd/popper.min.js">] (https://cdn.jsdelivr.net/npm/popper.js
W: memray: privacy-breach-generic [<script src="https://d3js.org/d3-scale-chromatic.v1.min.js">] (https://d3js.org/d3-scale-chromatic.v1.min.js) [usr/lib/pytho
W: memray: privacy-breach-generic [<script src="https://d3js.org/d3.v4.min.js" charset="utf-8">] (https://d3js.org/d3.v4.min.js) [usr/lib/python3/dist-packages
W: memray: python-module-in-wrong-location usr/lib/python3.12/dist-packages/memray-1.10.0.egg-info -> usr/lib/python3/dist-packages/memray-1.10.0.egg-info
W: memray source: space-in-std-shortname-in-dep5-copyright <special license> [debian/copyright:11]
W: memray: unusual-interpreter /usr/bin/python3.12 [usr/bin/memray3.12]

With 1.11.0 I fail completely with this build log: http://sid.ethz.ch/debian/memray/memray_1.11.0-1_amd64.build

Expected Behavior

1.11.0 should build, and 1.10.0 should have no warnings and errors

Steps To Reproduce

On Debian GNU/Linux with sid/unstable dget the source (*.dsc) and build it with debuild

Memray Version

1.10.0 and 1.11.0

Python Version

3.11

Operating System

Linux

Anything else?

No response

pablogsal commented 8 months ago

Hi @alexmyczko and thanks for opening the issue. Looking at the build failures I am seeing the following:

x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -g -O2 -ffile-prefix-map=/var/www/debian/memray/memray-1.11.0=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_FILE_OFFSET_BITS=64 -DUSE_MEMRAY_TLS_MODEL=1 -Isrc/memray/_memray -I/usr/include/x86_64-linux-gnu -Isrc -I/var/www/debian/memray/memray-1.11.0/src/vendor/libbacktrace/install/include -I/usr/include/python3.11 -c src/memray/_memray.cpp -o build/temp.linux-x86_64-cpython-311/src/memray/_memray.o -std=c++17 -Wall -flto -fvisibility=hidden
src/memray/_memray.cpp: In function ‘int __pyx_pf_6memray_7_memray_7Tracker___cinit__(__pyx_obj_6memray_7_memray_Tracker*, PyObject*, PyObject*, bool, unsigned int, bool, bool, __pyx_t_6memray_7_memray_FileFormat)’:
src/memray/_memray.cpp:11950:48: error: too few arguments to function ‘std::unique_ptr<memray::tracking_api::RecordWriter> memray::tracking_api::createRecordWriter(std::unique_ptr<memray::io::Sink>, const std::string&, bool, FileFormat, bool)’
11950 |     __pyx_t_8 = memray::api::createRecordWriter(cython_std::move<std::unique_ptr<memray::io::Sink> >(__pyx_t_7), __pyx_v_command_line, __pyx_v_native_traces, ((enum memray::tracking_api::FileFormat)__pyx_v_file_format));
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/memray/_memray.cpp:859:
src/memray/_memray/record_writer.h:61:1: note: declared here
   61 | createRecordWriter(
      | ^~~~~~~~~~~~~~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

This doesn't make any sense to me, it looks like you have not regenerated the Python sources. I can build memray locally without any problems and the CI can build it as well so this must be some problem in your Debian rules file or the way you are building the package. Unfortunately we cannot help more here unless you provide us with a reproducer that doesn't use any Debian infrastructure.

1.11.0 should build, and 1.10.0 should have no warnings and errors

Regarding the warnings, I am afraid we won't be able to help more here as these are aspects of how Debian wants to package projects. We are opened to considered Pull Requests but otherwise it looks like if you want to help some of these you will need to maintain your own patches.

pablogsal commented 8 months ago

For more details, notice that src/memray/_memray.cpp is generated during the build process and these are the lines generated from the latest version:

    __pyx_t_9 = memray::api::createRecordWriter(cython_std::move<std::unique_ptr<memray::io::Sink> >(__PYX_STD_MOVE_IF_SUPPORTED(__pyx_t_8)), __pyx_v_command_line, __pyx_v_native_traces, ((enum memray::tracking_api::FileFormat)__pyx_v_file_format), __pyx_v_trace_python_allocators);

as you can see there, the call is totally different and includes __pyx_v_trace_python_allocators which is not in your build logs. This hints that you are somehow producing an outdated Cython file in your build process.

alexmyczko commented 8 months ago

Hi @pablogsal that hint about src/memray/*.cpp was already helpful, thank you.

godlygeek commented 8 months ago

E: memray: arch-independent-package-contains-binary-or-object [usr/lib/python3/dist-packages/memray/_inject.abi3.so]

This error in particular indicates that you're trying to produce an architecture-independent package for Memray. That's wrong: memray includes native code, and needs to be compiled for each architecture. We support x86-64 (aka amd64) and aarch64 (aka arm64). So at least one problem that you've got here is that you've declared that the package should be architecture-independent, rather than architecture-dependent.

alexmyczko commented 8 months ago

The ITP link is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009970

alexmyczko commented 8 months ago

Ok now when I try to run it get this:

memray -h
Traceback (most recent call last):
  File "/usr/bin/memray", line 33, in <module>
    sys.exit(load_entry_point('memray==1.11.0', 'console_scripts', 'memray')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/memray", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/memray/__main__.py", line 3, in <module>
    from memray.commands import main
  File "/usr/lib/python3/dist-packages/memray/commands/__init__.py", line 19, in <module>
    from . import attach
  File "/usr/lib/python3/dist-packages/memray/commands/attach.py", line 19, in <module>
    from .live import LiveCommand
  File "/usr/lib/python3/dist-packages/memray/commands/live.py", line 7, in <module>
    from memray.reporters.tui import TUIApp
  File "/usr/lib/python3/dist-packages/memray/reporters/tui.py", line 30, in <module>
    from textual.app import ComposeResult
ImportError: cannot import name 'ComposeResult' from 'textual.app' (/usr/lib/python3/dist-packages/textual/app.py)
root@phd-sid:/var/www/debian/memray# ls -la /usr/lib/python3/dist-packages/textual/app.py
-rw-r--r-- 1 root root 18220 Jan  5  2022 /usr/lib/python3/dist-packages/textual/app.py
root@phd-sid:/var/www/debian/memray# dpkg -S /usr/lib/python3/dist-packages/textual/app.py
python3-textual: /usr/lib/python3/dist-packages/textual/app.py
root@phd-sid:/var/www/debian/memray# dpkg -l python3-textual
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version      Architecture Description
+++-===============-============-============-=================================================================================
ii  python3-textual 0.1.13-1     all          TUI (Text User Interface) framework for Python inspired by modern web development
pablogsal commented 8 months ago

This looks like it is because you seem to be using textual 0.1.13 while our requirement (check setup.py) is textual >= 0.34.0

alexmyczko commented 8 months ago

that is what i thought, what was confirmed yesterday on irc. and why i posted the version ;)

so trying 0.34.0 unfortunately that didn't help: ii python3-textual 0.34.0-0.1 all from https://sid.ethz.ch/debian/memray/python3-textual/

ok the error msg is a bit different

Traceback (most recent call last):
  File "/usr/bin/memray", line 33, in <module>
    sys.exit(load_entry_point('memray==1.11.0', 'console_scripts', 'memray')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/memray", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/memray/__main__.py", line 3, in <module>
    from memray.commands import main
  File "/usr/lib/python3/dist-packages/memray/commands/__init__.py", line 28, in <module>
    from . import tree
  File "/usr/lib/python3/dist-packages/memray/commands/tree.py", line 9, in <module>
    from memray.reporters.tree import TreeReporter
  File "/usr/lib/python3/dist-packages/memray/reporters/tree.py", line 33, in <module>
    from textual.widgets import TextArea
  File "/usr/lib/python3/dist-packages/textual/widgets/__init__.py", line 96, in __getattr__
    raise ImportError(f"Package 'textual.widgets' has no class '{widget_class}'")
ImportError: Package 'textual.widgets' has no class 'TextArea'
godlygeek commented 8 months ago

Ooh, that's a mistake on our part... it looks like we actually require at least Textual 0.41 now. Can you try with that version, @alexmyczko?

godlygeek commented 8 months ago

Alternatively, if you aren't able to upgrade Textual to (at least) 0.41 yet, you should be able to build and run Memray 1.10 (one version older than the newest) with Textual 0.34. The dependency on features that require Textual 0.41 came with Memray 1.11.

godlygeek commented 8 months ago

Correction: it looks like the minimum Textual version we support from Memray 1.11 on is now Textual 0.41, not Textual 0.39 (I've corrected my two comments above).

alexmyczko commented 8 months ago

0.41.0 is no problem: https://sid.ethz.ch/debian/memray/python3-textual/

can confirm memray runs now.

godlygeek commented 8 months ago

I don't think there's anything left to do for this particular issue.

Let me know if I'm wrong, @alexmyczko.

alexmyczko commented 7 months ago

well i'd really like to get this packaged for debian main, but i think i'll have to remove docs, tests, and src/memray/reporters will it still be useful for live/cli only mode? is building/running it that way supported somehow?

and i'd like to also de-vendor src/vendor/libbacktrace, is that supported? appears to be https://github.com/ianlancetaylor/libbacktrace not yet packaged https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008135

pablogsal commented 7 months ago

and src/memray/reporters will it still be useful for live/cli only mode?

Removing the reporters directory will make memray unusable and broken and is certainly not supported. Also, the live cli mode is a reporter as well. Please, do not package memray that way

and i'd like to also de-vendor src/vendor/libbacktrace, is that supported?

Not only it is not supported but it also won't work because we keep some patches on top of libbacktrace so it's not just a plain vendoring but a fork.

alexmyczko commented 7 months ago

perfect answer. so there must be a way to selfhost the html stuff. and thanks for the info on libbacktrace

mind if i keep the issue open? for visibility for others