Granulate / gprofiler

gProfiler is a system-wide profiler, combining multiple sampling profilers to produce unified visualization of what your CPU is spending time on.
https://profiler.granulate.io
Apache License 2.0
746 stars 54 forks source link

tests: py-spy fails on python 3.11 glibc on aarch64 #740

Closed pfilipko1 closed 1 year ago

pfilipko1 commented 1 year ago
application_pid = 1306515, assert_collapsed = functools.partial(<function assert_function_in_collapsed at 0xffffaa8ecd30>, 'burner'), profiler_type = 'py-spy', application_image_tag = '3.11-glibc-python' 
profiler_state = <gprofiler.profiler_state.ProfilerState object at 0xffffa8505cf0>                                                                                                                                                                                  [42/1986]                                                                                                                                                                                                                                                                                 @pytest.mark.parametrize("in_container", [True])                                                                                                                                                                                                                             @pytest.mark.parametrize(                                                                                                                                                                                                                                                        "application_image_tag",                                                                                                                                                                                                                                                     [                                                                                                                                                                                                                                                                                "2.7-glibc-python",                                                                                                                                                                                                                                                          "2.7-musl-python",                                                                                                                                                                                                                                                           "3.5-glibc-python",                                                                                                                                                                                                                                                          "3.5-musl-python",                                                                                                                                                                                                                                                           "3.6-glibc-python",                                                                                                                                                                                                                                                          "3.6-musl-python",                                                                                                                                                                                                                                                           "3.7-glibc-python",                                                                                                                                                                                                                                                          "3.7-musl-python",                                                                                                                                                                                                                                                           "3.8-glibc-python",                                                                                                                                                                                                                                                          "3.8-musl-python",                                                                                                                                                                                                                                                           "3.9-glibc-python",                                                                                                                                                                                                                                                          "3.9-musl-python",                                                                                                                                                                                                                                                           "3.10-glibc-python",                                                                                                                                                                                                                                                         "3.10-musl-python",                                                                                                                                                                                                                                                          "3.11-glibc-python",                                                                                                                                                                                                                                                         "3.11-musl-python",                                                                                                                                                                                                                                                          "2.7-glibc-uwsgi",                                                                                                                                                                                                                                                           "2.7-musl-uwsgi",                                                                                                                                                                                                                                                            "3.7-glibc-uwsgi",                                                                                                                                                                                                                                                           "3.7-musl-uwsgi",                                                                                                                                                                                                                                                        ],                                                                                                                                                                                                                                                                       )                                                                                                                                                                                                                                                                            @pytest.mark.parametrize("profiler_type", ["py-spy", "pyperf"])                                                                                                                                                                                                              def test_python_matrix(                                                                                                                                                                                                                                                          application_pid: int,                                                                                                                                                                                                                                                        assert_collapsed: AssertInCollapsed,                                                                                                                                                                                                                                         profiler_type: str,                                                                                                                                                                                                                                                          application_image_tag: str,                                                                                                                                                                                                                                                  profiler_state: ProfilerState,                                                                                                                                                                                                                                           ) -> None:                                                                                                                                                                                                                                                                       python_version, libc, app = application_image_tag.split("-")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              if python_version == "3.5" and profiler_type == "pyperf":                                                                                                                                                                                                                        pytest.skip("PyPerf doesn't support Python 3.5!")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     if python_version == "2.7" and profiler_type == "pyperf" and app == "uwsgi":                                                                                                                                                                                                     pytest.xfail("This combination fails, see https://github.com/Granulate/gprofiler/issues/485")                                                                                                                                                                    
        if python_version == "3.11" and profiler_type == "pyperf":
            pytest.xfail("PyPerf does not support 3.11 - https://github.com/Granulate/gprofiler/issues/727")

        if platform.machine() == "aarch64":                                                                                                                                                                                                                                              # pyperf is not working on aarch right now https://github.com/Granulate/gprofiler/issues/499                                                                                                                                                                                 if profiler_type == "pyperf":
                pytest.skip("PyPerf doesn't support aarch64 architecture!")

            if python_version == "2.7" and profiler_type == "py-spy" and app == "uwsgi":                                                                                                                                                                                                     pytest.xfail("This combination fails, see https://github.com/Granulate/gprofiler/issues/713")

            if python_version in ["3.7", "3.8", "3.9", "3.10", "3.11"] and profiler_type == "py-spy" and libc == "musl":
                pytest.xfail("This combination fails, see https://github.com/Granulate/gprofiler/issues/714")                                                                                                                                                                
        with PythonProfiler(1000, 2, profiler_state, profiler_type, True, None, False) as profiler:
            profile = snapshot_pid_profile(profiler, application_pid)

        collapsed = profile.stacks

>       assert_collapsed(collapsed)
tests/test_python.py:108:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
function_name = 'burner', collapsed = ProfilingErrorStack({'python;[Profiling error: exception CalledProcessError]': 1})
                                                                                                                                                                                                                                                                                 def assert_function_in_collapsed(function_name: str, collapsed: StackToSampleCount) -> None:                                                                                                                                                                                     print(f"collapsed: {collapsed}")
>       assert is_function_in_collapsed(function_name, collapsed), f"function {function_name!r} missing in collapsed data!"
E       AssertionError: function 'burner' missing in collapsed data!                                                                                                                                                                                                         
tests/utils.py:171: AssertionError
--------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------collapsed: ProfilingErrorStack({'python;[Profiling error: exception CalledProcessError]': 1})
----------------------------------------------------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------------------------------------------------WARNING  gprofiler.profilers.profiler_base:__init__.py:467 PySpyProfiler: Requested frequency (1000) is higher than the limit 50, limiting the frequency to the limit (50)                                                                                                   ERROR    gprofiler.profilers.profiler_base:profiler_base.py:145 PySpyProfiler: failed to profile process 1306515 (python)
Traceback (most recent call last):
  File "/home/ubuntu/gprofiler/gprofiler/profilers/profiler_base.py", line 132, in _wait_for_profiles                                                                                                                                                                            result = future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result                                                                                                                                                                                                    return self.__get_result()                                                                                                                                                                                                                                                 File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result                                                                                                                                                                                              raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run                                                                                                                                                                                                       result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                                                                                File "/home/ubuntu/gprofiler/gprofiler/profilers/python.py", line 218, in _profile_process                                                                                                                                                                                     run_process(
  File "/home/ubuntu/gprofiler/gprofiler/utils/__init__.py", line 284, in run_process
    raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
gprofiler.exceptions.CalledProcessError: Command ['/home/ubuntu/gprofiler/gprofiler/resources/python/py-spy', 'record', '-r', '50', '-d', '2', '--nonblocking', '--format', 'raw', '-F', '--output', '/tmp/pytest-of-root/pytest-67/test_python_matrix_py_spy_3_110/tmp35q3fsgl/pyspy.BQstSAwofIFeKidN.1306515.col', '-p', '1306515', '--full-filenames', '--gil'] returned non-zero exit status 1.
stdout: b''
stderr: b'Error: Unsupported version of Python: 3.11.0\n\n'
============================================================================================================================= warnings summary ==============================================================================================================================test_python.py: 91 warnings                                                                                                                                                                                                                                                    /usr/local/lib/python3.10/dist-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.                                                                                                                 s1 = StrictVersion(v1)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                test_python.py: 91 warnings                                                                                                                                                                                                                                                    /usr/local/lib/python3.10/dist-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.                                                                                                                 s2 = StrictVersion(v2)                                                                                                                                                                                                                                                   
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================================================== short test summary info ==========================================================================================================================FAILED tests/test_python.py::test_python_matrix[py-spy-3.11-glibc-python-True] - AssertionError: function 'burner' missing in collapsed data!****
Jongy commented 1 year ago

@pfilipko1 this is actually supported from https://github.com/Granulate/gprofiler/commit/ea29a9b668912691c04d96c56275afd5a0404543, try merging master into your branch and see?

Jongy commented 1 year ago

Closing since it does work since https://github.com/Granulate/gprofiler/commit/ea29a9b668912691c04d96c56275afd5a0404543.