search

bloomberg / memray

Memray is a memory profiler for Python
https://bloomberg.github.io/memray/
Apache License 2.0
13.19k stars 394 forks source link

ImportError: Package 'textual.widgets' has no class 'agm' #688

Closed ChrisDAT20 closed 2 hours ago

ChrisDAT20 commented 19 hours ago

Is there an existing issue for this?

Current Behavior

I want to profile a FastAPI based application, and use memray run -o memory.log /usr/local/bin/uvicorn ... to run the application. However, memray stops by throwing the following exception:

root@8593ec016ab9:/app# memray run -o memory.log /usr/local/bin/uvicorn app.main:app --host 0.0.0.0 --port 80
Writing profile results into memory.log
Memray WARNING: Correcting symbol for aligned_alloc from 0x7fe57d857d90 to 0x7fe57e0de550
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

/usr/local/bin/python -m memray flamegraph memory.log
Traceback (most recent call last):
  File "/usr/local/bin/memray", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/memray/commands/__init__.py", line 138, in main
    arg_values.entrypoint(arg_values, parser)
  File "/usr/local/lib/python3.9/site-packages/memray/commands/run.py", line 339, in run
    _run_with_file_output(args)
  File "/usr/local/lib/python3.9/site-packages/memray/commands/run.py", line 188, in _run_with_file_output
    _run_tracker(
  File "/usr/local/lib/python3.9/site-packages/memray/commands/run.py", line 77, in _run_tracker
    runpy.run_path(args.script, run_name="__main__")
  File "/usr/local/lib/python3.9/runpy.py", line 288, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/local/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 409, in main
    run(
  File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 575, in run
    server.run()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/simulation/app/main.py", line 8, in <module>
    import skops.io
  File "/usr/local/lib/python3.9/site-packages/skops/io/__init__.py", line 1, in <module>
    from ._persist import dump, dumps, get_untrusted_types, load, loads
  File "/usr/local/lib/python3.9/site-packages/skops/io/_persist.py", line 22, in <module>
    module = importlib.import_module(module_name, package="skops.io")
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.9/site-packages/skops/io/_general.py", line 16, in <module>
    from ._trusted_types import (
  File "/usr/local/lib/python3.9/site-packages/skops/io/_trusted_types.py", line 17, in <module>
    SCIPY_UFUNC_TYPE_NAMES = get_public_type_names(module=scipy.special, oftype=np.ufunc)
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 230, in get_public_type_names
    {
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 234, in <setcomp>
    and (type_name := get_type_name(obj)).startswith(module_name)
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 179, in get_type_name
    return f"{get_module(t)}.{t.__name__}"
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 86, in get_module
    return whichmodule(obj, obj.__name__)
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 49, in whichmodule
    if _getattribute(module, name)[0] is obj:
  File "/usr/local/lib/python3.9/site-packages/skops/io/_utils.py", line 24, in _getattribute
    obj = getattr(obj, subpath)
  File "/usr/local/lib/python3.9/site-packages/textual/widgets/__init__.py", line 107, in __getattr__
    raise ImportError(f"Package 'textual.widgets' has no class '{widget_class}'")
ImportError: Package 'textual.widgets' has no class 'agm'

I do not use the package textual not does the package skops.io. Why is that package imported? These versions are installed atm, does memray require another version?

root@8593ec016ab9:/app# pip install textual
Requirement already satisfied: textual in /usr/local/lib/python3.9/site-packages (0.81.0)
Requirement already satisfied: rich>=13.3.3 in /usr/local/lib/python3.9/site-packages (from textual) (13.9.0)
Requirement already satisfied: typing-extensions<5.0.0,>=4.4.0 in /usr/local/lib/python3.9/site-packages (from textual) (4.12.2)
Requirement already satisfied: platformdirs<5,>=3.6.0 in /usr/local/lib/python3.9/site-packages (from textual) (4.3.6)
Requirement already satisfied: markdown-it-py[linkify,plugins]>=2.1.0 in /usr/local/lib/python3.9/site-packages (from textual) (3.0.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/site-packages (from markdown-it-py[linkify,plugins]>=2.1.0->textual) (0.1.2)
Requirement already satisfied: mdit-py-plugins in /usr/local/lib/python3.9/site-packages (from markdown-it-py[linkify,plugins]>=2.1.0->textual) (0.4.2)
Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.9/site-packages (from markdown-it-py[linkify,plugins]>=2.1.0->textual) (2.0.3)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from rich>=13.3.3->textual) (2.18.0)
Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.9/site-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify,plugins]>=2.1.0->textual) (1.0.3)

Expected Behavior

memray does not lead to a import problem, when the observed does not have such issue.

Steps To Reproduce

see current behavior

Memray Version

1.14.0

Python Version

3.9

Operating System

Linux

Anything else?

Requirement already satisfied: memray in /usr/local/lib/python3.9/site-packages (1.14.0)
Requirement already satisfied: jinja2>=2.9 in /usr/local/lib/python3.9/site-packages (from memray) (3.1.4)
Requirement already satisfied: rich>=11.2.0 in /usr/local/lib/python3.9/site-packages (from memray) (13.9.0)
Requirement already satisfied: textual>=0.41.0 in /usr/local/lib/python3.9/site-packages (from memray) (0.81.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/site-packages (from jinja2>=2.9->memray) (2.1.5)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from rich>=11.2.0->memray) (2.18.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.9/site-packages (from rich>=11.2.0->memray) (3.0.0)
Requirement already satisfied: platformdirs<5,>=3.6.0 in /usr/local/lib/python3.9/site-packages (from textual>=0.41.0->memray) (4.3.6)
Requirement already satisfied: typing-extensions<5.0.0,>=4.4.0 in /usr/local/lib/python3.9/site-packages (from textual>=0.41.0->memray) (4.12.2)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=11.2.0->memray) (0.1.2)
Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=11.2.0->memray) (2.0.3)
Requirement already satisfied: mdit-py-plugins in /usr/local/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=11.2.0->memray) (0.4.2)
Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.9/site-packages (from linkify-it-py<3,>=1->markdown-it-py>=2.2.0->rich>=11.2.0->memray) (1.0.3)
godlygeek commented 19 hours ago

Memray depends on Textual, so that's why it's getting imported. skops is iterating over all the imported modules and trying to get an agm attribute from each. The textual.widgets module raises an ImportError rather than an AttributeError when you try to get an attribute from it that it doesn't have, and skops.io._utils is not prepared to deal with that exception. I think this is a bug in Textual, but it's certainly not a bug in Memray. I think this issue should be reported on the https://github.com/Textualize/textual repo instead.

ChrisDAT20 commented 18 hours ago

Ok. Somehow this ImportError has no effect when the app is executed outside of Memray.

godlygeek commented 18 hours ago

That's because the textual.widgets module is being imported by Memray, and not being imported when you're not running Memray. If you add an import textual.widgets to your code, you'll see this error even when not running Memray.

ChrisDAT20 commented 15 hours ago

Ok, thank you for your explanations. Is that agm attribute some convention which should be there?

godlygeek commented 11 hours ago

No, that seems to be something related to the working of the skops module. It seems to be trying to figure out which module provides an particular object, or something like that, and doing a linear scan over all the modules to see if they contain that object.

godlygeek commented 2 hours ago

Textual has fixed the bug, and the fix will be included in the next release.