search

pyutils / line_profiler

Line-by-line profiling for Python
Other
2.57k stars 118 forks source link

AttributeError after upgrading to line-profiler 4.0.0 from 3.5.1 #183

Closed Mogost closed 1 year ago

Mogost commented 1 year ago

I do not see anything in changelog that may cause this.

Traceback (most recent call last):
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/contrib/staticfiles/handlers.py", line 76, in __call__
    return self.application(environ, start_response)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
    response = self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/base.py", line 130, in get_response
    response = self._middleware_chain(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
    response = response_for_exception(request, exc)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/exception.py", line 114, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/exception.py", line 149, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django_extensions/management/technical_response.py", line 41, in null_technical_500_response
    raise exc_value
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/middleware.py", line 58, in __call__
    response = toolbar.process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/timer.py", line 65, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/headers.py", line 46, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  [Previous line repeated 1 more time]
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/template_profiler_panel/panels/template.py", line 250, in process_request
    response = super(TemplateProfilerPanel, self).process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/staticfiles.py", line 116, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/logging.py", line 95, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 206, in process_request
    self._unwrap_closure_and_profile(self.view_func)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 198, in _unwrap_closure_and_profile
    self._unwrap_closure_and_profile(value)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 198, in _unwrap_closure_and_profile
    self._unwrap_closure_and_profile(value)
  File "/Users/user/.pyenv/versions/v_ver/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 179, in _unwrap_closure_and_profile
    self.line_profiler.add_function(func)
  File "line_profiler/_line_profiler.pyx", line 196, in line_profiler._line_profiler.LineProfiler.add_function
  File "line_profiler/_line_profiler.pyx", line 219, in line_profiler._line_profiler.LineProfiler.add_function
AttributeError: 'method' object has no attribute '__code__'

Connected with https://github.com/mikekeda/django-debug-toolbar-line-profiler/issues/9

Erotemic commented 1 year ago

@Theelx can you take a look at this?

Theelx commented 1 year ago

Ah, I'll do some debugging now.

Theelx commented 1 year ago

@Mogost Can you provide some sample code to trigger this issue? I don't currently have my computer with me (I'm on my Github app now) so it'd help if I could see the code that's causing the issue and think about a solution while I'm on the move.

Theelx commented 1 year ago

I'm on my computer now, and when I add the @profile decorator to a method in a class, or when I create an instance of the LineProfiler class and call LineProfiler.add_function on the method, it works fine on CPython 3.8.13 and 3.9.13 (in pyenv). I see you're using 3.8, so that's interesting.

Theelx commented 1 year ago

I managed to reproduce it with this:

class Test:
    @profile
    @classmethod
    def method(self):
        print('hi')
Test.method()

That gives me this error:

/home/theel/.pyenv/versions/3.8.13/lib/python3.8/site-packages/line_profiler/line_profiler.py:51: UserWarning: Could not extract a code object for the object <classmethod object at 0x7fc05fb4e550>
  self.add_function(func)
Wrote profile results to test.py.lprof
Timer unit: 1e-06 s

Traceback (most recent call last):
  File "/home/theel/.pyenv/versions/3.8.13/bin/kernprof", line 8, in <module>
    sys.exit(main())
  File "/home/theel/.pyenv/versions/3.8.13/lib/python3.8/site-packages/kernprof.py", line 264, in main
    execfile(script_file, ns, ns)
  File "/home/theel/.pyenv/versions/3.8.13/lib/python3.8/site-packages/kernprof.py", line 32, in execfile
    exec(compile(f.read(), filename, 'exec'), globals, locals)
  File "test.py", line 3, in <module>
    class Test:
  File "test.py", line 6, in Test
    def method(self):
  File "/home/theel/.pyenv/versions/3.8.13/lib/python3.8/site-packages/line_profiler/line_profiler.py", line 54, in __call__
    elif is_generator(func):
  File "/home/theel/.pyenv/versions/3.8.13/lib/python3.8/site-packages/line_profiler/line_profiler.py", line 39, in is_generator
    isgen = (f.__code__.co_flags & CO_GENERATOR) != 0
AttributeError: 'classmethod' object has no attribute '__code__'

Luckily, there's a short-term fix:

class Test:
    @classmethod
    def method(self):
        print('hi')

Test.method = profile(Test.method)
Test.method()

Applying the profile decorator manually seems to somehow turn Test.method from a boundmethod to a function, which gives it the __code__ attribute.

hi
Wrote profile results to test.py.lprof
Timer unit: 1e-06 s

Total time: 2.061e-05 s
File: test.py
Function: method at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                               @classmethod
     5                                               def method(self):
     6         1         20.6     20.6    100.0          print('hi')

Edit: I'm still going to do my best to fix this issue, however since there's a short-term fix, I need to prioritize my classwork and studying (I'm in college!) Hopefully, I can get a fix within 2-3 days.

Mogost commented 1 year ago

I can't really provide an example because I use https://github.com/mikekeda/django-debug-toolbar-line-profiler/ which is in turn a plugin for https://github.com/jazzband/django-debug-toolbar/ on a big Django project. All I have is an exception trace.

It is definitely worth paying attention to the plugin itself, which I use.

Theelx commented 1 year ago

@Mogost Can you try using the Test.method = profile(Test.method) trick that I mentioned above, and see if it works?

Mogost commented 1 year ago 
Traceback (most recent call last):
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/contrib/staticfiles/handlers.py", line 76, in __call__
    return self.application(environ, start_response)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
    response = self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/base.py", line 130, in get_response
    response = self._middleware_chain(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
    response = response_for_exception(request, exc)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/exception.py", line 114, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/exception.py", line 149, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django_extensions/management/technical_response.py", line 41, in null_technical_500_response
    raise exc_value
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/middleware.py", line 58, in __call__
    response = toolbar.process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/timer.py", line 65, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/headers.py", line 46, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  [Previous line repeated 1 more time]
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/template_profiler_panel/panels/template.py", line 250, in process_request
    response = super(TemplateProfilerPanel, self).process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/staticfiles.py", line 116, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/logging.py", line 95, in process_request
    return super().process_request(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 206, in process_request
    return self.get_response(request)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 206, in process_request
    self._unwrap_closure_and_profile(self.view_func)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 198, in _unwrap_closure_and_profile
    self._unwrap_closure_and_profile(value)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 198, in _unwrap_closure_and_profile
    self._unwrap_closure_and_profile(value)
  File "/Users/user/.pyenv/versions/var/lib/python3.8/site-packages/debug_toolbar_line_profiler/panel.py", line 179, in _unwrap_closure_and_profile
    self.line_profiler.add_function(func)
  File "line_profiler/_line_profiler.pyx", line 196, in line_profiler._line_profiler.LineProfiler.add_function
  File "line_profiler/_line_profiler.pyx", line 221, in line_profiler._line_profiler.LineProfiler.add_function
AttributeError: 'method' object has no attribute '__code__'

The problem still persists in 4.0.1

Theelx commented 1 year ago

Does using the profile() trick work?

cliffckerr commented 1 year ago

I'm also having this issue, although it's hard to reproduce. It caused my GitHub Actions tests to fail: https://github.com/sciris/sciris/actions/runs/3615173776/jobs/6092102445

../sciris/sc_profiling.py:370: in profile
    wrapper = lp(run)
/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/line_profiler/line_profiler.py:53: in __call__
    self.add_function(func)
line_profiler/_line_profiler.pyx:196: in line_profiler._line_profiler.LineProfiler.add_function
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   AttributeError: 'method' object has no attribute '__code__'

line_profiler/_line_profiler.pyx:221: AttributeError

This use case wraps LineProfiler by calling LineProfiler.add_function(): https://github.com/sciris/sciris/blob/v2.0.4/sciris/sc_profiling.py#L365

What's weird is that I can't reproduce this bug locally -- I have a fresh conda environment with everything seemingly the same as on GitHub, but the test passes.

cliffckerr commented 1 year ago

I'm not very experienced reading Cython code, so this could be a red herring, but the change I saw that seems most likely to be related to this bug is the change in the old version of the code from

<object>py_frame.f_code

being used to initialize the LineTiming class, to

code = func.__code__

being used in the new _code_replace() instead.

Theelx commented 1 year ago

I'm looking into this now with your new information. Thanks for reporting this!

Theelx commented 1 year ago

@cliffckerr It should be fixed in #191. Can you confirm that https://github.com/Theelx/sciris/actions/runs/3616146614 fixes the error and looks good before I ask @Erotemic to merge my PR?

cliffckerr commented 1 year ago

Yes that looks great -- thank you so much for the fast response @Theelx !

Mogost commented 1 year ago

Fixed by #191