SystemProfiler: ensure samples are collected from event

roi-granulate commented 4 months ago

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots

Checklist:

[ ] Please post on the PR an example log displaying how it'd look when this new check fail (you can inject a fault to the code to make sure it does break, for example remove the "ok" return)
[ ] Sanity test on aarch64

roi-granulate commented 4 months ago

@Jongy attaching the log of a failed run (I have not this condition on a "healthy" machine)

$ sudo ./build/x86_64/gprofiler --output-dir=/tmp/
[2024-02-28 17:23:17,662] INFO: gprofiler: Running gProfiler (version=1.45.0, commandline=--output-dir=/tmp/, arguments={'pid_file': '/var/run/gprofiler.pid', 'config': None, 'frequency': 11, 'duration': 60, 'insert_dso_name': False, 'output_dir': '/tmp/', 'flamegraph': True, 'profiling_mode': 'cpu', 'alloc_interval': None, 'rotating_output': False, 'pids_to_profile': None, 'dotnet_mode': 'disabled', 'python_mode': 'auto', 'python_add_versions': True, 'python_pyperf_user_stacks_pages': None, 'python_pyperf_verbose': False, 'java_mode': 'ap', 'java_version_check': True, 'java_async_profiler_mode': 'auto', 'java_async_profiler_safemode': 0, 'java_async_profiler_features': ['probesp', 'vtable'], 'java_async_profiler_args': None, 'java_safemode': 'profiled-oom,profiled-signaled,hserr', 'java_jattach_timeout': 30, 'java_async_profiler_mcache': 30, 'java_collect_spark_app_name_as_appid': False, 'java_async_profiler_report_meminfo': True, 'java_collect_jvm_flags': 'default', 'java_full_hserr': False, 'java_include_method_modifiers': False, 'java_line_numbers': 'none', 'perf_mode': 'fp', 'perf_dwarf_stack_size': 8192, 'perf_memory_restart': True, 'php_mode': 'disabled', 'php_process_filter': 'php-fpm', 'ruby_mode': 'rbspy', 'spark_sample_period': 120, 'collect_spark_metrics': False, 'nodejs_mode': 'disabled', 'log_usage': False, 'upload_results': False, 'disable_core_files': True, 'subcommand': None, 'server_host': 'https://profiler.granulate.io', 'api_server': 'https://api.granulate.io', 'server_upload_timeout': 120, 'server_token': None, 'service_name': None, 'curlify_requests': False, 'verify': True, 'verbose': False, 'log_file': '/var/log/gprofiler/gprofiler.log', 'log_rotate_max_size': 5242880, 'log_rotate_backup_count': 1, 'log_to_server': True, 'container_names': True, 'continuous': False, 'profile_api_version': None, 'pid_ns_check': True, 'collect_metrics': True, 'collect_metadata': True, 'collect_appids': True, 'app_id_args_filters': [], 'application_metadata': True, 'controller_pid': None, 'external_metadata': None, 'databricks_job_name_as_service_name': False, 'profile_spawned_processes': False, 'diagnostics': False, 'perf_inject': False, 'perf_node_attach': False, 'extract_resources': False})
[2024-02-28 17:23:17,831] INFO: gprofiler: gProfiler Python version: 3.10.13 (main, Feb 21 2024, 15:25:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
[2024-02-28 17:23:17,832] INFO: gprofiler: gProfiler deployment mode: standalone_executable
[2024-02-28 17:23:17,832] INFO: gprofiler: Kernel uname release: 6.5.0
[2024-02-28 17:23:17,832] INFO: gprofiler: Kernel uname version: #14~22.04.1-Ubuntu SMP Thu Feb 15 15:27:06 UTC 2024
[2024-02-28 17:23:17,833] INFO: gprofiler: Total CPUs: 8
[2024-02-28 17:23:17,833] INFO: gprofiler: Total RAM: 31.33 GB
[2024-02-28 17:23:17,833] INFO: gprofiler: Linux distribution: Ubuntu | 22.04 | Jammy Jellyfish
[2024-02-28 17:23:17,833] INFO: gprofiler: libc version: glibc-2.35-0ubuntu3.6
[2024-02-28 17:23:17,834] INFO: gprofiler: Hostname: hostname
[2024-02-28 17:23:17,895] INFO: gprofiler.containers_client: Discovered container runtimes: ['docker']
[2024-02-28 17:23:17,896] INFO: gprofiler.profilers.profiler_base: Initialized PythonEbpfProfiler (frequency: 11hz, duration: 60s), profiling mode: cpu
[2024-02-28 17:23:24,134] INFO: gprofiler.profilers.profiler_base: Initialized JavaProfiler (frequency: 11hz, duration: 60s), profiling mode: cpu
[2024-02-28 17:23:28,619] INFO: gprofiler.profilers.profiler_base: Initialized SystemProfiler (frequency: 11hz, duration: 60s), profiling mode: cpu
[2024-02-28 17:23:28,620] INFO: gprofiler.utils.perf_process: Starting perf (fp mode)
[2024-02-28 17:23:28,726] INFO: gprofiler.utils.perf_process: Started perf (fp mode)
[2024-02-28 17:23:30,749] INFO: gprofiler.utils.perf_process: Stopped perf (fp mode) (exit_code=0, stderr=b'', stdout=b'')
[2024-02-28 17:23:30,749] INFO: gprofiler.utils.perf_process: Starting perf (fp mode)
[2024-02-28 17:23:30,856] INFO: gprofiler.utils.perf_process: Started perf (fp mode)
[2024-02-28 17:23:32,197] INFO: gprofiler.utils.perf_process: Stopped perf (fp mode) (exit_code=0, stderr=b'', stdout=b'')
[2024-02-28 17:23:32,198] INFO: gprofiler.utils.perf_process: Starting perf (fp mode)
[2024-02-28 17:23:32,304] INFO: gprofiler.utils.perf_process: Started perf (fp mode)
[2024-02-28 17:23:33,659] INFO: gprofiler.utils.perf_process: Stopped perf (fp mode) (exit_code=0, stderr=b'', stdout=b'')
[2024-02-28 17:23:33,659] CRITICAL: gprofiler.profilers.perf: Failed to determine perf event to use
[2024-02-28 17:23:33,660] CRITICAL: gprofiler.profilers.factory: Couldn't create the Perf profiler, not continuing. Run with --no-perf to disable this profiler
Traceback (most recent call last):
  File "gprofiler/profilers/factory.py", line 54, in get_profilers
    profiler_instance = profiler_config.profiler_class(**profiler_kwargs)
  File "gprofiler/profilers/perf.py", line 185, in __init__
    get_perf_event_args(Path(self._profiler_state.storage_dir), self._profiler_state.stop_event)
  File "gprofiler/utils/perf.py", line 104, in get_perf_event_args
    raise PerfNoSupportedEvent
gprofiler.exceptions.PerfNoSupportedEvent

Jongy commented 4 months ago

Looks fine - I'll merge after CI with merged master passes, and we'll release 1.46.0 tmrw.

Granulate / gprofiler

SystemProfiler: ensure samples are collected from event #887