Support AMD Piledriver?

[edwintorok]

I wanted to try rr but it looks like it doesn't support my AMD CPU:

[FATAL /home/edwin/rr/rr/src/PerfCounters.cc:129:get_cpu_microarch() errno: 0 'Success'] 
 -> CPU 0xf20 unknown.

The CPU is an AMD FX-8350, which is AFAICT Piledriver microarchitecture, see below for cpuinfo and perf list output. The available performance counter events should be documented in 3.15 Core Performance Counter Events. Is it possible to support rr on this microarchitecture, and what would be the value for struct PmuConfig?

$ dmesg|grep PMU
[    0.346218] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    0.346626] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
$ uname -a
Linux debian 4.2.0-0.bpo.1-amd64 #1 SMP Debian 4.2.3-2~bpo8+1 (2015-10-20) x86_64 GNU/Linux

/proc/cpuinfo for a core:

processor   : 7
vendor_id   : AuthenticAMD
cpu family  : 21
model       : 2
model name  : AMD FX(tm)-8350 Eight-Core Processor
stepping    : 0
microcode   : 0x6000832
cpu MHz     : 1400.000
cache size  : 2048 KB
physical id : 0
siblings    : 8
core id     : 7
cpu cores   : 4
apicid      : 23
initial apicid  : 7
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 8026.66
TLB size    : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

List of pre-defined events (to be used in -e):

branch-instructions OR branches [Hardware event] branch-misses [Hardware event] cache-misses [Hardware event] cache-references [Hardware event] cpu-cycles OR cycles [Hardware event] instructions [Hardware event] stalled-cycles-backend OR idle-cycles-backend [Hardware event] stalled-cycles-frontend OR idle-cycles-frontend [Hardware event]

alignment-faults [Software event] context-switches OR cs [Software event] cpu-clock [Software event] cpu-migrations OR migrations [Software event] dummy [Software event] emulation-faults [Software event] major-faults [Software event] minor-faults [Software event] page-faults OR faults [Software event] task-clock [Software event]

L1-dcache-load-misses [Hardware cache event] L1-dcache-loads [Hardware cache event] L1-dcache-prefetch-misses [Hardware cache event] L1-dcache-prefetches [Hardware cache event] L1-dcache-stores [Hardware cache event] L1-icache-load-misses [Hardware cache event] L1-icache-loads [Hardware cache event] L1-icache-prefetches [Hardware cache event] LLC-load-misses [Hardware cache event] LLC-loads [Hardware cache event] LLC-stores [Hardware cache event] branch-load-misses [Hardware cache event] branch-loads [Hardware cache event] dTLB-load-misses [Hardware cache event] dTLB-loads [Hardware cache event] iTLB-load-misses [Hardware cache event] iTLB-loads [Hardware cache event]

branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] branch-misses OR cpu/branch-misses/ [Kernel PMU event] cache-misses OR cpu/cache-misses/ [Kernel PMU event] cache-references OR cpu/cache-references/ [Kernel PMU event] cpu-cycles OR cpu/cpu-cycles/ [Kernel PMU event] instructions OR cpu/instructions/ [Kernel PMU event]

rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor](see 'man perf-list' on how to encode it)

mem:[/len][:access] [Hardware breakpoint]

block:block_bio_backmerge [Tracepoint event] block:block_bio_bounce [Tracepoint event] block:block_bio_complete [Tracepoint event] block:block_bio_frontmerge [Tracepoint event] block:block_bio_queue [Tracepoint event] block:block_bio_remap [Tracepoint event] block:block_dirty_buffer [Tracepoint event] block:block_getrq [Tracepoint event] block:block_plug [Tracepoint event] block:block_rq_abort [Tracepoint event] block:block_rq_complete [Tracepoint event] block:block_rq_insert [Tracepoint event] block:block_rq_issue [Tracepoint event] block:block_rq_remap [Tracepoint event] block:block_rq_requeue [Tracepoint event] block:block_sleeprq [Tracepoint event] block:block_split [Tracepoint event] block:block_touch_buffer [Tracepoint event] block:block_unplug [Tracepoint event] btrfs:__extent_writepage [Tracepoint event] btrfs:add_delayed_data_ref [Tracepoint event] btrfs:add_delayed_ref_head [Tracepoint event] btrfs:add_delayed_tree_ref [Tracepoint event] btrfs:alloc_extent_state [Tracepoint event] btrfs:btrfs_all_work_done [Tracepoint event] btrfs:btrfs_chunk_alloc [Tracepoint event] btrfs:btrfs_chunk_free [Tracepoint event] btrfs:btrfs_cow_block [Tracepoint event] btrfs:btrfs_failed_cluster_setup [Tracepoint event] btrfs:btrfs_find_cluster [Tracepoint event] btrfs:btrfs_get_extent [Tracepoint event] btrfs:btrfs_inode_evict [Tracepoint event] btrfs:btrfs_inode_new [Tracepoint event] btrfs:btrfs_inode_request [Tracepoint event] btrfs:btrfs_ordered_extent_add [Tracepoint event] btrfs:btrfs_ordered_extent_put [Tracepoint event] btrfs:btrfs_ordered_extent_remove [Tracepoint event] btrfs:btrfs_ordered_extent_start [Tracepoint event] btrfs:btrfs_ordered_sched [Tracepoint event] btrfs:btrfs_reserve_extent [Tracepoint event] btrfs:btrfs_reserve_extent_cluster [Tracepoint event] btrfs:btrfs_reserved_extent_alloc [Tracepoint event] btrfs:btrfs_reserved_extent_free [Tracepoint event] btrfs:btrfs_setup_cluster [Tracepoint event] btrfs:btrfs_space_reservation [Tracepoint event] btrfs:btrfs_sync_file [Tracepoint event] btrfs:btrfs_sync_fs [Tracepoint event] btrfs:btrfs_transaction_commit [Tracepoint event] btrfs:btrfs_work_queued [Tracepoint event] btrfs:btrfs_work_sched [Tracepoint event] btrfs:btrfs_workqueue_alloc [Tracepoint event] btrfs:btrfs_workqueue_destroy [Tracepoint event] btrfs:btrfs_writepage_end_io_hook [Tracepoint event] btrfs:find_free_extent [Tracepoint event] btrfs:free_extent_state [Tracepoint event] btrfs:run_delayed_data_ref [Tracepoint event] btrfs:run_delayed_ref_head [Tracepoint event] btrfs:run_delayed_tree_ref [Tracepoint event] cfg80211:cfg80211_cac_event [Tracepoint event] cfg80211:cfg80211_ch_switch_notify [Tracepoint event] cfg80211:cfg80211_ch_switch_started_notify [Tracepoint event] cfg80211:cfg80211_chandef_dfs_required [Tracepoint event] cfg80211:cfg80211_cqm_pktloss_notify [Tracepoint event] cfg80211:cfg80211_cqm_rssi_notify [Tracepoint event] cfg80211:cfg80211_del_sta [Tracepoint event] cfg80211:cfg80211_ft_event [Tracepoint event] cfg80211:cfg80211_get_bss [Tracepoint event] cfg80211:cfg80211_gtk_rekey_notify [Tracepoint event] cfg80211:cfg80211_ibss_joined [Tracepoint event] cfg80211:cfg80211_inform_bss_width_frame [Tracepoint event] cfg80211:cfg80211_mgmt_tx_status [Tracepoint event] cfg80211:cfg80211_michael_mic_failure [Tracepoint event] cfg80211:cfg80211_new_sta [Tracepoint event] cfg80211:cfg80211_notify_new_peer_candidate [Tracepoint event] cfg80211:cfg80211_pmksa_candidate_notify [Tracepoint event] cfg80211:cfg80211_probe_status [Tracepoint event] cfg80211:cfg80211_radar_event [Tracepoint event] cfg80211:cfg80211_ready_on_channel [Tracepoint event] cfg80211:cfg80211_ready_on_channel_expired [Tracepoint event] cfg80211:cfg80211_reg_can_beacon [Tracepoint event] cfg80211:cfg80211_report_obss_beacon [Tracepoint event] cfg80211:cfg80211_report_wowlan_wakeup [Tracepoint event] cfg80211:cfg80211_return_bool [Tracepoint event] cfg80211:cfg80211_return_bss [Tracepoint event] cfg80211:cfg80211_return_u32 [Tracepoint event] cfg80211:cfg80211_return_uint [Tracepoint event] cfg80211:cfg80211_rx_mgmt [Tracepoint event] cfg80211:cfg80211_rx_mlme_mgmt [Tracepoint event] cfg80211:cfg80211_rx_spurious_frame [Tracepoint event] cfg80211:cfg80211_rx_unexpected_4addr_frame [Tracepoint event] cfg80211:cfg80211_rx_unprot_mlme_mgmt [Tracepoint event] cfg80211:cfg80211_scan_done [Tracepoint event] cfg80211:cfg80211_sched_scan_results [Tracepoint event] cfg80211:cfg80211_sched_scan_stopped [Tracepoint event] cfg80211:cfg80211_send_assoc_timeout [Tracepoint event] cfg80211:cfg80211_send_auth_timeout [Tracepoint event] cfg80211:cfg80211_send_rx_assoc [Tracepoint event] cfg80211:cfg80211_send_rx_auth [Tracepoint event] cfg80211:cfg80211_stop_iface [Tracepoint event] cfg80211:cfg80211_tdls_oper_request [Tracepoint event] cfg80211:cfg80211_tx_mlme_mgmt [Tracepoint event] cfg80211:rdev_add_key [Tracepoint event] cfg80211:rdev_add_mpath [Tracepoint event] cfg80211:rdev_add_station [Tracepoint event] cfg80211:rdev_add_tx_ts [Tracepoint event] cfg80211:rdev_add_virtual_intf [Tracepoint event] cfg80211:rdev_assoc [Tracepoint event] cfg80211:rdev_auth [Tracepoint event] cfg80211:rdev_cancel_remain_on_channel [Tracepoint event] cfg80211:rdev_change_beacon [Tracepoint event] cfg80211:rdev_change_bss [Tracepoint event] cfg80211:rdev_change_mpath [Tracepoint event] cfg80211:rdev_change_station [Tracepoint event] cfg80211:rdev_change_virtual_intf [Tracepoint event] cfg80211:rdev_channel_switch [Tracepoint event] cfg80211:rdev_connect [Tracepoint event] cfg80211:rdev_crit_proto_start [Tracepoint event] cfg80211:rdev_crit_proto_stop [Tracepoint event] cfg80211:rdev_deauth [Tracepoint event] cfg80211:rdev_del_key [Tracepoint event] cfg80211:rdev_del_mpath [Tracepoint event] cfg80211:rdev_del_pmksa [Tracepoint event] cfg80211:rdev_del_station [Tracepoint event] cfg80211:rdev_del_tx_ts [Tracepoint event] cfg80211:rdev_del_virtual_intf [Tracepoint event] cfg80211:rdev_disassoc [Tracepoint event] cfg80211:rdev_disconnect [Tracepoint event] cfg80211:rdev_dump_mpath [Tracepoint event] cfg80211:rdev_dump_mpp [Tracepoint event] cfg80211:rdev_dump_station [Tracepoint event] cfg80211:rdev_dump_survey [Tracepoint event] cfg80211:rdev_flush_pmksa [Tracepoint event] cfg80211:rdev_get_antenna [Tracepoint event] cfg80211:rdev_get_channel [Tracepoint event] cfg80211:rdev_get_key [Tracepoint event] cfg80211:rdev_get_mesh_config [Tracepoint event] cfg80211:rdev_get_mpath [Tracepoint event] cfg80211:rdev_get_mpp [Tracepoint event] cfg80211:rdev_get_station [Tracepoint event] cfg80211:rdev_get_tx_power [Tracepoint event] cfg80211:rdev_join_ibss [Tracepoint event] cfg80211:rdev_join_mesh [Tracepoint event] cfg80211:rdev_join_ocb [Tracepoint event] cfg80211:rdev_leave_ibss [Tracepoint event] cfg80211:rdev_leave_mesh [Tracepoint event] cfg80211:rdev_leave_ocb [Tracepoint event] cfg80211:rdev_libertas_set_mesh_channel [Tracepoint event] cfg80211:rdev_mgmt_frame_register [Tracepoint event] cfg80211:rdev_mgmt_tx [Tracepoint event] cfg80211:rdev_mgmt_tx_cancel_wait [Tracepoint event] cfg80211:rdev_probe_client [Tracepoint event] cfg80211:rdev_remain_on_channel [Tracepoint event] cfg80211:rdev_resume [Tracepoint event] cfg80211:rdev_return_chandef [Tracepoint event] cfg80211:rdev_return_int [Tracepoint event] cfg80211:rdev_return_int_cookie [Tracepoint event] cfg80211:rdev_return_int_int [Tracepoint event] cfg80211:rdev_return_int_mesh_config [Tracepoint event] cfg80211:rdev_return_int_mpath_info [Tracepoint event] cfg80211:rdev_return_int_station_info [Tracepoint event] cfg80211:rdev_return_int_survey_info [Tracepoint event] cfg80211:rdev_return_int_tx_rx [Tracepoint event] cfg80211:rdev_return_void [Tracepoint event] cfg80211:rdev_return_void_tx_rx [Tracepoint event] cfg80211:rdev_return_wdev [Tracepoint event] cfg80211:rdev_rfkill_poll [Tracepoint event] cfg80211:rdev_scan [Tracepoint event] cfg80211:rdev_sched_scan_start [Tracepoint event] cfg80211:rdev_sched_scan_stop [Tracepoint event] cfg80211:rdev_set_antenna [Tracepoint event] cfg80211:rdev_set_ap_chanwidth [Tracepoint event] cfg80211:rdev_set_bitrate_mask [Tracepoint event] cfg80211:rdev_set_cqm_rssi_config [Tracepoint event] cfg80211:rdev_set_cqm_txe_config [Tracepoint event] cfg80211:rdev_set_default_key [Tracepoint event] cfg80211:rdev_set_default_mgmt_key [Tracepoint event] cfg80211:rdev_set_mac_acl [Tracepoint event] cfg80211:rdev_set_monitor_channel [Tracepoint event] cfg80211:rdev_set_noack_map [Tracepoint event] cfg80211:rdev_set_pmksa [Tracepoint event] cfg80211:rdev_set_power_mgmt [Tracepoint event] cfg80211:rdev_set_qos_map [Tracepoint event] cfg80211:rdev_set_rekey_data [Tracepoint event] cfg80211:rdev_set_tx_power [Tracepoint event] cfg80211:rdev_set_txq_params [Tracepoint event] cfg80211:rdev_set_wakeup [Tracepoint event] cfg80211:rdev_set_wds_peer [Tracepoint event] cfg80211:rdev_set_wiphy_params [Tracepoint event] cfg80211:rdev_start_ap [Tracepoint event] cfg80211:rdev_start_p2p_device [Tracepoint event] cfg80211:rdev_stop_ap [Tracepoint event] cfg80211:rdev_stop_p2p_device [Tracepoint event] cfg80211:rdev_suspend [Tracepoint event] cfg80211:rdev_tdls_cancel_channel_switch [Tracepoint event] cfg80211:rdev_tdls_channel_switch [Tracepoint event] cfg80211:rdev_tdls_mgmt [Tracepoint event] cfg80211:rdev_tdls_oper [Tracepoint event] cfg80211:rdev_update_ft_ies [Tracepoint event] cfg80211:rdev_update_mesh_config [Tracepoint event] clk:clk_disable [Tracepoint event] clk:clk_disable_complete [Tracepoint event] clk:clk_enable [Tracepoint event] clk:clk_enable_complete [Tracepoint event] clk:clk_prepare [Tracepoint event] clk:clk_prepare_complete [Tracepoint event] clk:clk_set_parent [Tracepoint event] clk:clk_set_parent_complete [Tracepoint event] clk:clk_set_phase [Tracepoint event] clk:clk_set_phase_complete [Tracepoint event] clk:clk_set_rate [Tracepoint event] clk:clk_set_rate_complete [Tracepoint event] clk:clk_unprepare [Tracepoint event] clk:clk_unprepare_complete [Tracepoint event] compaction:mm_compaction_begin [Tracepoint event] compaction:mm_compaction_defer_compaction [Tracepoint event] compaction:mm_compaction_defer_reset [Tracepoint event] compaction:mm_compaction_deferred [Tracepoint event] compaction:mm_compaction_end [Tracepoint event] compaction:mm_compaction_finished [Tracepoint event] compaction:mm_compaction_isolate_freepages [Tracepoint event] compaction:mm_compaction_isolate_migratepages [Tracepoint event] compaction:mm_compaction_migratepages [Tracepoint event] compaction:mm_compaction_suitable [Tracepoint event] compaction:mm_compaction_try_to_compact_pages [Tracepoint event] drm:drm_vblank_event [Tracepoint event] drm:drm_vblank_event_delivered [Tracepoint event] drm:drm_vblank_event_queued [Tracepoint event] exceptions:page_fault_kernel [Tracepoint event] exceptions:page_fault_user [Tracepoint event] ext4:ext4_alloc_da_blocks [Tracepoint event] ext4:ext4_allocate_blocks [Tracepoint event] ext4:ext4_allocate_inode [Tracepoint event] ext4:ext4_begin_ordered_truncate [Tracepoint event] ext4:ext4_collapse_range [Tracepoint event] ext4:ext4_da_release_space [Tracepoint event] ext4:ext4_da_reserve_space [Tracepoint event] ext4:ext4_da_update_reserve_space [Tracepoint event] ext4:ext4_da_write_begin [Tracepoint event] ext4:ext4_da_write_end [Tracepoint event] ext4:ext4_da_write_pages [Tracepoint event] ext4:ext4_da_write_pages_extent [Tracepoint event] ext4:ext4_direct_IO_enter [Tracepoint event] ext4:ext4_direct_IO_exit [Tracepoint event] ext4:ext4_discard_blocks [Tracepoint event] ext4:ext4_discard_preallocations [Tracepoint event] ext4:ext4_drop_inode [Tracepoint event] ext4:ext4_es_cache_extent [Tracepoint event] ext4:ext4_es_find_delayed_extent_range_enter [Tracepoint event] ext4:ext4_es_find_delayed_extent_range_exit [Tracepoint event] ext4:ext4_es_insert_extent [Tracepoint event] ext4:ext4_es_lookup_extent_enter [Tracepoint event] ext4:ext4_es_lookup_extent_exit [Tracepoint event] ext4:ext4_es_remove_extent [Tracepoint event] ext4:ext4_es_shrink [Tracepoint event] ext4:ext4_es_shrink_count [Tracepoint event] ext4:ext4_es_shrink_scan_enter [Tracepoint event] ext4:ext4_es_shrink_scan_exit [Tracepoint event] ext4:ext4_evict_inode [Tracepoint event] ext4:ext4_ext_convert_to_initialized_enter [Tracepoint event] ext4:ext4_ext_convert_to_initialized_fastpath [Tracepoint event] ext4:ext4_ext_handle_unwritten_extents [Tracepoint event] ext4:ext4_ext_in_cache [Tracepoint event] ext4:ext4_ext_load_extent [Tracepoint event] ext4:ext4_ext_map_blocks_enter [Tracepoint event] ext4:ext4_ext_map_blocks_exit [Tracepoint event] ext4:ext4_ext_put_in_cache [Tracepoint event] ext4:ext4_ext_remove_space [Tracepoint event] ext4:ext4_ext_remove_space_done [Tracepoint event] ext4:ext4_ext_rm_idx [Tracepoint event] ext4:ext4_ext_rm_leaf [Tracepoint event] ext4:ext4_ext_show_extent [Tracepoint event] ext4:ext4_fallocate_enter [Tracepoint event] ext4:ext4_fallocate_exit [Tracepoint event] ext4:ext4_find_delalloc_range [Tracepoint event] ext4:ext4_forget [Tracepoint event] ext4:ext4_free_blocks [Tracepoint event] ext4:ext4_free_inode [Tracepoint event] ext4:ext4_get_implied_cluster_alloc_exit [Tracepoint event] ext4:ext4_get_reserved_cluster_alloc [Tracepoint event] ext4:ext4_ind_map_blocks_enter [Tracepoint event] ext4:ext4_ind_map_blocks_exit [Tracepoint event] ext4:ext4_insert_range [Tracepoint event] ext4:ext4_invalidatepage [Tracepoint event] ext4:ext4_journal_start [Tracepoint event] ext4:ext4_journal_start_reserved [Tracepoint event] ext4:ext4_journalled_invalidatepage [Tracepoint event] ext4:ext4_journalled_write_end [Tracepoint event] ext4:ext4_load_inode [Tracepoint event] ext4:ext4_load_inode_bitmap [Tracepoint event] ext4:ext4_mark_inode_dirty [Tracepoint event] ext4:ext4_mb_bitmap_load [Tracepoint event] ext4:ext4_mb_buddy_bitmap_load [Tracepoint event] ext4:ext4_mb_discard_preallocations [Tracepoint event] ext4:ext4_mb_new_group_pa [Tracepoint event] ext4:ext4_mb_new_inode_pa [Tracepoint event] ext4:ext4_mb_release_group_pa [Tracepoint event] ext4:ext4_mb_release_inode_pa [Tracepoint event] ext4:ext4_mballoc_alloc [Tracepoint event] ext4:ext4_mballoc_discard [Tracepoint event] ext4:ext4_mballoc_free [Tracepoint event] ext4:ext4_mballoc_prealloc [Tracepoint event] ext4:ext4_other_inode_update_time [Tracepoint event] ext4:ext4_punch_hole [Tracepoint event] ext4:ext4_read_block_bitmap_load [Tracepoint event] ext4:ext4_readpage [Tracepoint event] ext4:ext4_releasepage [Tracepoint event] ext4:ext4_remove_blocks [Tracepoint event] ext4:ext4_request_blocks [Tracepoint event] ext4:ext4_request_inode [Tracepoint event] ext4:ext4_sync_file_enter [Tracepoint event] ext4:ext4_sync_file_exit [Tracepoint event] ext4:ext4_sync_fs [Tracepoint event] ext4:ext4_trim_all_free [Tracepoint event] ext4:ext4_trim_extent [Tracepoint event] ext4:ext4_truncate_enter [Tracepoint event] ext4:ext4_truncate_exit [Tracepoint event] ext4:ext4_unlink_enter [Tracepoint event] ext4:ext4_unlink_exit [Tracepoint event] ext4:ext4_write_begin [Tracepoint event] ext4:ext4_write_end [Tracepoint event] ext4:ext4_writepage [Tracepoint event] ext4:ext4_writepages [Tracepoint event] ext4:ext4_writepages_result [Tracepoint event] ext4:ext4_zero_range [Tracepoint event] fence:fence_annotate_wait_on [Tracepoint event] fence:fence_destroy [Tracepoint event] fence:fence_emit [Tracepoint event] fence:fence_enable_signal [Tracepoint event] fence:fence_init [Tracepoint event] fence:fence_signaled [Tracepoint event] fence:fence_wait_end [Tracepoint event] fence:fence_wait_start [Tracepoint event] filelock:break_lease_block [Tracepoint event] filelock:break_lease_noblock [Tracepoint event] filelock:break_lease_unblock [Tracepoint event] filelock:generic_add_lease [Tracepoint event] filelock:generic_delete_lease [Tracepoint event] filelock:time_out_leases [Tracepoint event] filemap:mm_filemap_add_to_page_cache [Tracepoint event] filemap:mm_filemap_delete_from_page_cache [Tracepoint event] ftrace:function [Tracepoint event] gpio:gpio_direction [Tracepoint event] gpio:gpio_value [Tracepoint event] hda:hda_get_response [Tracepoint event] hda:hda_send_cmd [Tracepoint event] hda:hda_unsol_event [Tracepoint event] hda:snd_hdac_stream_start [Tracepoint event] hda:snd_hdac_stream_stop [Tracepoint event] hda_controller:azx_get_position [Tracepoint event] hda_controller:azx_pcm_close [Tracepoint event] hda_controller:azx_pcm_hw_params [Tracepoint event] hda_controller:azx_pcm_open [Tracepoint event] hda_controller:azx_pcm_prepare [Tracepoint event] hda_controller:azx_pcm_trigger [Tracepoint event] hda_intel:azx_resume [Tracepoint event] hda_intel:azx_runtime_resume [Tracepoint event] hda_intel:azx_runtime_suspend [Tracepoint event] hda_intel:azx_suspend [Tracepoint event] i2c:i2c_read [Tracepoint event] i2c:i2c_reply [Tracepoint event] i2c:i2c_result [Tracepoint event] i2c:i2c_write [Tracepoint event] i2c:smbus_read [Tracepoint event] i2c:smbus_reply [Tracepoint event] i2c:smbus_result [Tracepoint event] i2c:smbus_write [Tracepoint event] iommu:add_device_to_group [Tracepoint event] iommu:attach_device_to_domain [Tracepoint event] iommu:detach_device_from_domain [Tracepoint event] iommu:io_page_fault [Tracepoint event] iommu:map [Tracepoint event] iommu:remove_device_from_group [Tracepoint event] iommu:unmap [Tracepoint event] irq:irq_handler_entry [Tracepoint event] irq:irq_handler_exit [Tracepoint event] irq:softirq_entry [Tracepoint event] irq:softirq_exit [Tracepoint event] irq:softirq_raise [Tracepoint event] irq_vectors:call_function_entry [Tracepoint event] irq_vectors:call_function_exit [Tracepoint event] irq_vectors:call_function_single_entry [Tracepoint event] irq_vectors:call_function_single_exit [Tracepoint event] irq_vectors:deferred_error_apic_entry [Tracepoint event] irq_vectors:deferred_error_apic_exit [Tracepoint event] irq_vectors:error_apic_entry [Tracepoint event] irq_vectors:error_apic_exit [Tracepoint event] irq_vectors:irq_work_entry [Tracepoint event] irq_vectors:irq_work_exit [Tracepoint event] irq_vectors:local_timer_entry [Tracepoint event] irq_vectors:local_timer_exit [Tracepoint event] irq_vectors:reschedule_entry [Tracepoint event] irq_vectors:reschedule_exit [Tracepoint event] irq_vectors:spurious_apic_entry [Tracepoint event] irq_vectors:spurious_apic_exit [Tracepoint event] irq_vectors:thermal_apic_entry [Tracepoint event] irq_vectors:thermal_apic_exit [Tracepoint event] irq_vectors:threshold_apic_entry [Tracepoint event] irq_vectors:threshold_apic_exit [Tracepoint event] irq_vectors:x86_platform_ipi_entry [Tracepoint event] irq_vectors:x86_platform_ipi_exit [Tracepoint event] jbd2:jbd2_checkpoint [Tracepoint event] jbd2:jbd2_checkpoint_stats [Tracepoint event] jbd2:jbd2_commit_flushing [Tracepoint event] jbd2:jbd2_commit_locking [Tracepoint event] jbd2:jbd2_commit_logging [Tracepoint event] jbd2:jbd2_drop_transaction [Tracepoint event] jbd2:jbd2_end_commit [Tracepoint event] jbd2:jbd2_handle_extend [Tracepoint event] jbd2:jbd2_handle_start [Tracepoint event] jbd2:jbd2_handle_stats [Tracepoint event] jbd2:jbd2_lock_buffer_stall [Tracepoint event] jbd2:jbd2_run_stats [Tracepoint event] jbd2:jbd2_start_commit [Tracepoint event] jbd2:jbd2_submit_inode_data [Tracepoint event] jbd2:jbd2_update_log_tail [Tracepoint event] jbd2:jbd2_write_superblock [Tracepoint event] kmem:kfree [Tracepoint event] kmem:kmalloc [Tracepoint event] kmem:kmalloc_node [Tracepoint event] kmem:kmem_cache_alloc [Tracepoint event] kmem:kmem_cache_alloc_node [Tracepoint event] kmem:kmem_cache_free [Tracepoint event] kmem:mm_page_alloc [Tracepoint event] kmem:mm_page_alloc_extfrag [Tracepoint event] kmem:mm_page_alloc_zone_locked [Tracepoint event] kmem:mm_page_free [Tracepoint event] kmem:mm_page_free_batched [Tracepoint event] kmem:mm_page_pcpu_drain [Tracepoint event] kvm:kvm_ack_irq [Tracepoint event] kvm:kvm_age_page [Tracepoint event] kvm:kvm_apic [Tracepoint event] kvm:kvm_apic_accept_irq [Tracepoint event] kvm:kvm_apic_ipi [Tracepoint event] kvm:kvm_async_pf_completed [Tracepoint event] kvm:kvm_async_pf_doublefault [Tracepoint event] kvm:kvm_async_pf_not_present [Tracepoint event] kvm:kvm_async_pf_ready [Tracepoint event] kvm:kvm_cpuid [Tracepoint event] kvm:kvm_cr [Tracepoint event] kvm:kvm_emulate_insn [Tracepoint event] kvm:kvm_enter_smm [Tracepoint event] kvm:kvm_entry [Tracepoint event] kvm:kvm_eoi [Tracepoint event] kvm:kvm_exit [Tracepoint event] kvm:kvm_fpu [Tracepoint event] kvm:kvm_hv_hypercall [Tracepoint event] kvm:kvm_hypercall [Tracepoint event] kvm:kvm_inj_exception [Tracepoint event] kvm:kvm_inj_virq [Tracepoint event] kvm:kvm_invlpga [Tracepoint event] kvm:kvm_ioapic_delayed_eoi_inj [Tracepoint event] kvm:kvm_ioapic_set_irq [Tracepoint event] kvm:kvm_mmio [Tracepoint event] kvm:kvm_msi_set_irq [Tracepoint event] kvm:kvm_msr [Tracepoint event] kvm:kvm_nested_intercepts [Tracepoint event] kvm:kvm_nested_intr_vmexit [Tracepoint event] kvm:kvm_nested_vmexit [Tracepoint event] kvm:kvm_nested_vmexit_inject [Tracepoint event] kvm:kvm_nested_vmrun [Tracepoint event] kvm:kvm_page_fault [Tracepoint event] kvm:kvm_pic_set_irq [Tracepoint event] kvm:kvm_pio [Tracepoint event] kvm:kvm_ple_window [Tracepoint event] kvm:kvm_pml_full [Tracepoint event] kvm:kvm_pv_eoi [Tracepoint event] kvm:kvm_pvclock_update [Tracepoint event] kvm:kvm_set_irq [Tracepoint event] kvm:kvm_skinit [Tracepoint event] kvm:kvm_track_tsc [Tracepoint event] kvm:kvm_try_async_get_page [Tracepoint event] kvm:kvm_update_master_clock [Tracepoint event] kvm:kvm_userspace_exit [Tracepoint event] kvm:kvm_vcpu_wakeup [Tracepoint event] kvm:kvm_wait_lapic_expire [Tracepoint event] kvm:kvm_write_tsc_offset [Tracepoint event] kvm:vcpu_match_mmio [Tracepoint event] kvmmmu:check_mmio_spte [Tracepoint event] kvmmmu:fast_page_fault [Tracepoint event] kvmmmu:handle_mmio_page_fault [Tracepoint event] kvmmmu:kvm_mmu_get_page [Tracepoint event] kvmmmu:kvm_mmu_invalidate_zap_all_pages [Tracepoint event] kvmmmu:kvm_mmu_pagetable_walk [Tracepoint event] kvmmmu:kvm_mmu_paging_element [Tracepoint event] kvmmmu:kvm_mmu_prepare_zap_page [Tracepoint event] kvmmmu:kvm_mmu_set_accessed_bit [Tracepoint event] kvmmmu:kvm_mmu_set_dirty_bit [Tracepoint event] kvmmmu:kvm_mmu_sync_page [Tracepoint event] kvmmmu:kvm_mmu_unsync_page [Tracepoint event] kvmmmu:kvm_mmu_walker_error [Tracepoint event] kvmmmu:mark_mmio_spte [Tracepoint event] libata:ata_eh_link_autopsy [Tracepoint event] libata:ata_eh_link_autopsy_qc [Tracepoint event] libata:ata_qc_complete_done [Tracepoint event] libata:ata_qc_complete_failed [Tracepoint event] libata:ata_qc_complete_internal [Tracepoint event] libata:ata_qc_issue [Tracepoint event] mce:mce_record [Tracepoint event] migrate:mm_migrate_pages [Tracepoint event] migrate:mm_numa_migrate_ratelimit [Tracepoint event] module:module_free [Tracepoint event] module:module_get [Tracepoint event] module:module_load [Tracepoint event] module:module_put [Tracepoint event] module:module_request [Tracepoint event] napi:napi_poll [Tracepoint event] net:napi_gro_frags_entry [Tracepoint event] net:napi_gro_receive_entry [Tracepoint event] net:net_dev_queue [Tracepoint event] net:net_dev_start_xmit [Tracepoint event] net:net_dev_xmit [Tracepoint event] net:netif_receive_skb [Tracepoint event] net:netif_receive_skb_entry [Tracepoint event] net:netif_rx [Tracepoint event] net:netif_rx_entry [Tracepoint event] net:netif_rx_ni_entry [Tracepoint event] nmi:nmi_handler [Tracepoint event] oom:oom_score_adj_update [Tracepoint event] pagemap:mm_lru_activate [Tracepoint event] pagemap:mm_lru_insertion [Tracepoint event] power:clock_disable [Tracepoint event] power:clock_enable [Tracepoint event] power:clock_set_rate [Tracepoint event] power:cpu_frequency [Tracepoint event] power:cpu_idle [Tracepoint event] power:dev_pm_qos_add_request [Tracepoint event] power:dev_pm_qos_remove_request [Tracepoint event] power:dev_pm_qos_update_request [Tracepoint event] power:device_pm_callback_end [Tracepoint event] power:device_pm_callback_start [Tracepoint event] power:pm_qos_add_request [Tracepoint event] power:pm_qos_remove_request [Tracepoint event] power:pm_qos_update_flags [Tracepoint event] power:pm_qos_update_request [Tracepoint event] power:pm_qos_update_request_timeout [Tracepoint event] power:pm_qos_update_target [Tracepoint event] power:power_domain_target [Tracepoint event] power:pstate_sample [Tracepoint event] power:suspend_resume [Tracepoint event] power:wakeup_source_activate [Tracepoint event] power:wakeup_source_deactivate [Tracepoint event] printk:console [Tracepoint event] radeon:radeon_bo_create [Tracepoint event] radeon:radeon_cs [Tracepoint event] radeon:radeon_fence_emit [Tracepoint event] radeon:radeon_fence_wait_begin [Tracepoint event] radeon:radeon_fence_wait_end [Tracepoint event] radeon:radeon_semaphore_signale [Tracepoint event] radeon:radeon_semaphore_wait [Tracepoint event] radeon:radeon_vm_bo_update [Tracepoint event] radeon:radeon_vm_flush [Tracepoint event] radeon:radeon_vm_grab_id [Tracepoint event] radeon:radeon_vm_set_page [Tracepoint event] random:add_device_randomness [Tracepoint event] random:add_disk_randomness [Tracepoint event] random:add_input_randomness [Tracepoint event] random:credit_entropy_bits [Tracepoint event] random:debit_entropy [Tracepoint event] random:extract_entropy [Tracepoint event] random:extract_entropy_user [Tracepoint event] random:get_random_bytes [Tracepoint event] random:get_random_bytes_arch [Tracepoint event] random:mix_pool_bytes [Tracepoint event] random:mix_pool_bytes_nolock [Tracepoint event] random:push_to_pool [Tracepoint event] random:random_read [Tracepoint event] random:urandom_read [Tracepoint event] random:xfer_secondary_pool [Tracepoint event] ras:aer_event [Tracepoint event] ras:extlog_mem_event [Tracepoint event] ras:mc_event [Tracepoint event] ras:memory_failure_event [Tracepoint event] raw_syscalls:sys_enter [Tracepoint event] raw_syscalls:sys_exit [Tracepoint event] rcu:rcu_utilization [Tracepoint event] regmap:regcache_drop_region [Tracepoint event] regmap:regcache_sync [Tracepoint event] regmap:regmap_async_complete_done [Tracepoint event] regmap:regmap_async_complete_start [Tracepoint event] regmap:regmap_async_io_complete [Tracepoint event] regmap:regmap_async_write_start [Tracepoint event] regmap:regmap_cache_bypass [Tracepoint event] regmap:regmap_cache_only [Tracepoint event] regmap:regmap_hw_read_done [Tracepoint event] regmap:regmap_hw_read_start [Tracepoint event] regmap:regmap_hw_write_done [Tracepoint event] regmap:regmap_hw_write_start [Tracepoint event] regmap:regmap_reg_read [Tracepoint event] regmap:regmap_reg_read_cache [Tracepoint event] regmap:regmap_reg_write [Tracepoint event] rpm:rpm_idle [Tracepoint event] rpm:rpm_resume [Tracepoint event] rpm:rpm_return_int [Tracepoint event] rpm:rpm_suspend [Tracepoint event] sched:sched_kthread_stop [Tracepoint event] sched:sched_kthread_stop_ret [Tracepoint event] sched:sched_migrate_task [Tracepoint event] sched:sched_move_numa [Tracepoint event] sched:sched_pi_setprio [Tracepoint event] sched:sched_process_exec [Tracepoint event] sched:sched_process_exit [Tracepoint event] sched:sched_process_fork [Tracepoint event] sched:sched_process_free [Tracepoint event] sched:sched_process_hang [Tracepoint event] sched:sched_process_wait [Tracepoint event] sched:sched_stat_blocked [Tracepoint event] sched:sched_stat_iowait [Tracepoint event] sched:sched_stat_runtime [Tracepoint event] sched:sched_stat_sleep [Tracepoint event] sched:sched_stat_wait [Tracepoint event] sched:sched_stick_numa [Tracepoint event] sched:sched_swap_numa [Tracepoint event] sched:sched_switch [Tracepoint event] sched:sched_wait_task [Tracepoint event] sched:sched_wake_idle_without_ipi [Tracepoint event] sched:sched_wakeup [Tracepoint event] sched:sched_wakeup_new [Tracepoint event] scsi:scsi_dispatch_cmd_done [Tracepoint event] scsi:scsi_dispatch_cmd_error [Tracepoint event] scsi:scsi_dispatch_cmd_start [Tracepoint event] scsi:scsi_dispatch_cmd_timeout [Tracepoint event] scsi:scsi_eh_wakeup [Tracepoint event] signal:signal_deliver [Tracepoint event] signal:signal_generate [Tracepoint event] skb:consume_skb [Tracepoint event] skb:kfree_skb [Tracepoint event] skb:skb_copy_datagram_iovec [Tracepoint event] sock:sock_exceed_buf_limit [Tracepoint event] sock:sock_rcvqueue_full [Tracepoint event] spi:spi_master_busy [Tracepoint event] spi:spi_master_idle [Tracepoint event] spi:spi_message_done [Tracepoint event] spi:spi_message_start [Tracepoint event] spi:spi_message_submit [Tracepoint event] spi:spi_transfer_start [Tracepoint event] spi:spi_transfer_stop [Tracepoint event] swiotlb:swiotlb_bounced [Tracepoint event] syscalls:sys_enter_accept [Tracepoint event] syscalls:sys_enter_accept4 [Tracepoint event] syscalls:sys_enter_access [Tracepoint event] syscalls:sys_enter_acct [Tracepoint event] syscalls:sys_enter_add_key [Tracepoint event] syscalls:sys_enter_adjtimex [Tracepoint event] syscalls:sys_enter_alarm [Tracepoint event] syscalls:sys_enter_bind [Tracepoint event] syscalls:sys_enter_brk [Tracepoint event] syscalls:sys_enter_capget [Tracepoint event] syscalls:sys_enter_capset [Tracepoint event] syscalls:sys_enter_chdir [Tracepoint event] syscalls:sys_enter_chmod [Tracepoint event] syscalls:sys_enter_chown [Tracepoint event] syscalls:sys_enter_chroot [Tracepoint event] syscalls:sys_enter_clock_adjtime [Tracepoint event] syscalls:sys_enter_clock_getres [Tracepoint event] syscalls:sys_enter_clock_gettime [Tracepoint event] syscalls:sys_enter_clock_nanosleep [Tracepoint event] syscalls:sys_enter_clock_settime [Tracepoint event] syscalls:sys_enter_close [Tracepoint event] syscalls:sys_enter_connect [Tracepoint event] syscalls:sys_enter_creat [Tracepoint event] syscalls:sys_enter_delete_module [Tracepoint event] syscalls:sys_enter_dup [Tracepoint event] syscalls:sys_enter_dup2 [Tracepoint event] syscalls:sys_enter_dup3 [Tracepoint event] syscalls:sys_enter_epoll_create [Tracepoint event] syscalls:sys_enter_epoll_create1 [Tracepoint event] syscalls:sys_enter_epoll_ctl [Tracepoint event] syscalls:sys_enter_epoll_pwait [Tracepoint event] syscalls:sys_enter_epoll_wait [Tracepoint event] syscalls:sys_enter_eventfd [Tracepoint event] syscalls:sys_enter_eventfd2 [Tracepoint event] syscalls:sys_enter_exit [Tracepoint event] syscalls:sys_enter_exit_group [Tracepoint event] syscalls:sys_enter_faccessat [Tracepoint event] syscalls:sys_enter_fadvise64 [Tracepoint event] syscalls:sys_enter_fallocate [Tracepoint event] syscalls:sys_enter_fanotify_init [Tracepoint event] syscalls:sys_enter_fanotify_mark [Tracepoint event] syscalls:sys_enter_fchdir [Tracepoint event] syscalls:sys_enter_fchmod [Tracepoint event] syscalls:sys_enter_fchmodat [Tracepoint event] syscalls:sys_enter_fchown [Tracepoint event] syscalls:sys_enter_fchownat [Tracepoint event] syscalls:sys_enter_fcntl [Tracepoint event] syscalls:sys_enter_fdatasync [Tracepoint event] syscalls:sys_enter_fgetxattr [Tracepoint event] syscalls:sys_enter_finit_module [Tracepoint event] syscalls:sys_enter_flistxattr [Tracepoint event] syscalls:sys_enter_flock [Tracepoint event] syscalls:sys_enter_fremovexattr [Tracepoint event] syscalls:sys_enter_fsetxattr [Tracepoint event] syscalls:sys_enter_fstatfs [Tracepoint event] syscalls:sys_enter_fsync [Tracepoint event] syscalls:sys_enter_ftruncate [Tracepoint event] syscalls:sys_enter_futex [Tracepoint event] syscalls:sys_enter_futimesat [Tracepoint event] syscalls:sys_enter_get_mempolicy [Tracepoint event] syscalls:sys_enter_get_robust_list [Tracepoint event] syscalls:sys_enter_getcpu [Tracepoint event] syscalls:sys_enter_getcwd [Tracepoint event] syscalls:sys_enter_getdents [Tracepoint event] syscalls:sys_enter_getdents64 [Tracepoint event] syscalls:sys_enter_getegid [Tracepoint event] syscalls:sys_enter_geteuid [Tracepoint event] syscalls:sys_enter_getgid [Tracepoint event] syscalls:sys_enter_getgroups [Tracepoint event] syscalls:sys_enter_getitimer [Tracepoint event] syscalls:sys_enter_getpeername [Tracepoint event] syscalls:sys_enter_getpgid [Tracepoint event] syscalls:sys_enter_getpgrp [Tracepoint event] syscalls:sys_enter_getpid [Tracepoint event] syscalls:sys_enter_getppid [Tracepoint event] syscalls:sys_enter_getpriority [Tracepoint event] syscalls:sys_enter_getrandom [Tracepoint event] syscalls:sys_enter_getresgid [Tracepoint event] syscalls:sys_enter_getresuid [Tracepoint event] syscalls:sys_enter_getrlimit [Tracepoint event] syscalls:sys_enter_getrusage [Tracepoint event] syscalls:sys_enter_getsid [Tracepoint event] syscalls:sys_enter_getsockname [Tracepoint event] syscalls:sys_enter_getsockopt [Tracepoint event] syscalls:sys_enter_gettid [Tracepoint event] syscalls:sys_enter_gettimeofday [Tracepoint event] syscalls:sys_enter_getuid [Tracepoint event] syscalls:sys_enter_getxattr [Tracepoint event] syscalls:sys_enter_init_module [Tracepoint event] syscalls:sys_enter_inotify_add_watch [Tracepoint event] syscalls:sys_enter_inotify_init [Tracepoint event] syscalls:sys_enter_inotify_init1 [Tracepoint event] syscalls:sys_enter_inotify_rm_watch [Tracepoint event] syscalls:sys_enter_io_cancel [Tracepoint event] syscalls:sys_enter_io_destroy [Tracepoint event] syscalls:sys_enter_io_getevents [Tracepoint event] syscalls:sys_enter_io_setup [Tracepoint event] syscalls:sys_enter_io_submit [Tracepoint event] syscalls:sys_enter_ioctl [Tracepoint event] syscalls:sys_enter_iopl [Tracepoint event] syscalls:sys_enter_ioprio_get [Tracepoint event] syscalls:sys_enter_ioprio_set [Tracepoint event] syscalls:sys_enter_kcmp [Tracepoint event] syscalls:sys_enter_kexec_load [Tracepoint event] syscalls:sys_enter_keyctl [Tracepoint event] syscalls:sys_enter_kill [Tracepoint event] syscalls:sys_enter_lchown [Tracepoint event] syscalls:sys_enter_lgetxattr [Tracepoint event] syscalls:sys_enter_link [Tracepoint event] syscalls:sys_enter_linkat [Tracepoint event] syscalls:sys_enter_listen [Tracepoint event] syscalls:sys_enter_listxattr [Tracepoint event] syscalls:sys_enter_llistxattr [Tracepoint event] syscalls:sys_enter_lookup_dcookie [Tracepoint event] syscalls:sys_enter_lremovexattr [Tracepoint event] syscalls:sys_enter_lseek [Tracepoint event] syscalls:sys_enter_lsetxattr [Tracepoint event] syscalls:sys_enter_madvise [Tracepoint event] syscalls:sys_enter_mbind [Tracepoint event] syscalls:sys_enter_memfd_create [Tracepoint event] syscalls:sys_enter_migrate_pages [Tracepoint event] syscalls:sys_enter_mincore [Tracepoint event] syscalls:sys_enter_mkdir [Tracepoint event] syscalls:sys_enter_mkdirat [Tracepoint event] syscalls:sys_enter_mknod [Tracepoint event] syscalls:sys_enter_mknodat [Tracepoint event] syscalls:sys_enter_mlock [Tracepoint event] syscalls:sys_enter_mlockall [Tracepoint event] syscalls:sys_enter_mmap [Tracepoint event] syscalls:sys_enter_mount [Tracepoint event] syscalls:sys_enter_move_pages [Tracepoint event] syscalls:sys_enter_mprotect [Tracepoint event] syscalls:sys_enter_mq_getsetattr [Tracepoint event] syscalls:sys_enter_mq_notify [Tracepoint event] syscalls:sys_enter_mq_open [Tracepoint event] syscalls:sys_enter_mq_timedreceive [Tracepoint event] syscalls:sys_enter_mq_timedsend [Tracepoint event] syscalls:sys_enter_mq_unlink [Tracepoint event] syscalls:sys_enter_mremap [Tracepoint event] syscalls:sys_enter_msgctl [Tracepoint event] syscalls:sys_enter_msgget [Tracepoint event] syscalls:sys_enter_msgrcv [Tracepoint event] syscalls:sys_enter_msgsnd [Tracepoint event] syscalls:sys_enter_msync [Tracepoint event] syscalls:sys_enter_munlock [Tracepoint event] syscalls:sys_enter_munlockall [Tracepoint event] syscalls:sys_enter_munmap [Tracepoint event] syscalls:sys_enter_name_to_handle_at [Tracepoint event] syscalls:sys_enter_nanosleep [Tracepoint event] syscalls:sys_enter_newfstat [Tracepoint event] syscalls:sys_enter_newfstatat [Tracepoint event] syscalls:sys_enter_newlstat [Tracepoint event] syscalls:sys_enter_newstat [Tracepoint event] syscalls:sys_enter_newuname [Tracepoint event] syscalls:sys_enter_open [Tracepoint event] syscalls:sys_enter_open_by_handle_at [Tracepoint event] syscalls:sys_enter_openat [Tracepoint event] syscalls:sys_enter_pause [Tracepoint event] syscalls:sys_enter_perf_event_open [Tracepoint event] syscalls:sys_enter_personality [Tracepoint event] syscalls:sys_enter_pipe [Tracepoint event] syscalls:sys_enter_pipe2 [Tracepoint event] syscalls:sys_enter_pivot_root [Tracepoint event] syscalls:sys_enter_poll [Tracepoint event] syscalls:sys_enter_ppoll [Tracepoint event] syscalls:sys_enter_prctl [Tracepoint event] syscalls:sys_enter_pread64 [Tracepoint event] syscalls:sys_enter_preadv [Tracepoint event] syscalls:sys_enter_prlimit64 [Tracepoint event] syscalls:sys_enter_process_vm_readv [Tracepoint event] syscalls:sys_enter_process_vm_writev [Tracepoint event] syscalls:sys_enter_pselect6 [Tracepoint event] syscalls:sys_enter_ptrace [Tracepoint event] syscalls:sys_enter_pwrite64 [Tracepoint event] syscalls:sys_enter_pwritev [Tracepoint event] syscalls:sys_enter_quotactl [Tracepoint event] syscalls:sys_enter_read [Tracepoint event] syscalls:sys_enter_readahead [Tracepoint event] syscalls:sys_enter_readlink [Tracepoint event] syscalls:sys_enter_readlinkat [Tracepoint event] syscalls:sys_enter_readv [Tracepoint event] syscalls:sys_enter_reboot [Tracepoint event] syscalls:sys_enter_recvfrom [Tracepoint event] syscalls:sys_enter_recvmmsg [Tracepoint event] syscalls:sys_enter_recvmsg [Tracepoint event] syscalls:sys_enter_remap_file_pages [Tracepoint event] syscalls:sys_enter_removexattr [Tracepoint event] syscalls:sys_enter_rename [Tracepoint event] syscalls:sys_enter_renameat [Tracepoint event] syscalls:sys_enter_renameat2 [Tracepoint event] syscalls:sys_enter_request_key [Tracepoint event] syscalls:sys_enter_restart_syscall [Tracepoint event] syscalls:sys_enter_rmdir [Tracepoint event] syscalls:sys_enter_rt_sigaction [Tracepoint event] syscalls:sys_enter_rt_sigpending [Tracepoint event] syscalls:sys_enter_rt_sigprocmask [Tracepoint event] syscalls:sys_enter_rt_sigqueueinfo [Tracepoint event] syscalls:sys_enter_rt_sigsuspend [Tracepoint event] syscalls:sys_enter_rt_sigtimedwait [Tracepoint event] syscalls:sys_enter_rt_tgsigqueueinfo [Tracepoint event] syscalls:sys_enter_sched_get_priority_max [Tracepoint event] syscalls:sys_enter_sched_get_priority_min [Tracepoint event] syscalls:sys_enter_sched_getaffinity [Tracepoint event] syscalls:sys_enter_sched_getattr [Tracepoint event] syscalls:sys_enter_sched_getparam [Tracepoint event] syscalls:sys_enter_sched_getscheduler [Tracepoint event] syscalls:sys_enter_sched_rr_get_interval [Tracepoint event] syscalls:sys_enter_sched_setaffinity [Tracepoint event] syscalls:sys_enter_sched_setattr [Tracepoint event] syscalls:sys_enter_sched_setparam [Tracepoint event] syscalls:sys_enter_sched_setscheduler [Tracepoint event] syscalls:sys_enter_sched_yield [Tracepoint event] syscalls:sys_enter_seccomp [Tracepoint event] syscalls:sys_enter_select [Tracepoint event] syscalls:sys_enter_semctl [Tracepoint event] syscalls:sys_enter_semget [Tracepoint event] syscalls:sys_enter_semop [Tracepoint event] syscalls:sys_enter_semtimedop [Tracepoint event] syscalls:sys_enter_sendfile64 [Tracepoint event] syscalls:sys_enter_sendmmsg [Tracepoint event] syscalls:sys_enter_sendmsg [Tracepoint event] syscalls:sys_enter_sendto [Tracepoint event] syscalls:sys_enter_set_mempolicy [Tracepoint event] syscalls:sys_enter_set_robust_list [Tracepoint event] syscalls:sys_enter_set_tid_address [Tracepoint event] syscalls:sys_enter_setdomainname [Tracepoint event] syscalls:sys_enter_setfsgid [Tracepoint event] syscalls:sys_enter_setfsuid [Tracepoint event] syscalls:sys_enter_setgid [Tracepoint event] syscalls:sys_enter_setgroups [Tracepoint event] syscalls:sys_enter_sethostname [Tracepoint event] syscalls:sys_enter_setitimer [Tracepoint event] syscalls:sys_enter_setns [Tracepoint event] syscalls:sys_enter_setpgid [Tracepoint event] syscalls:sys_enter_setpriority [Tracepoint event] syscalls:sys_enter_setregid [Tracepoint event] syscalls:sys_enter_setresgid [Tracepoint event] syscalls:sys_enter_setresuid [Tracepoint event] syscalls:sys_enter_setreuid [Tracepoint event] syscalls:sys_enter_setrlimit [Tracepoint event] syscalls:sys_enter_setsid [Tracepoint event] syscalls:sys_enter_setsockopt [Tracepoint event] syscalls:sys_enter_settimeofday [Tracepoint event] syscalls:sys_enter_setuid [Tracepoint event] syscalls:sys_enter_setxattr [Tracepoint event] syscalls:sys_enter_shmat [Tracepoint event] syscalls:sys_enter_shmctl [Tracepoint event] syscalls:sys_enter_shmdt [Tracepoint event] syscalls:sys_enter_shmget [Tracepoint event] syscalls:sys_enter_shutdown [Tracepoint event] syscalls:sys_enter_sigaltstack [Tracepoint event] syscalls:sys_enter_signalfd [Tracepoint event] syscalls:sys_enter_signalfd4 [Tracepoint event] syscalls:sys_enter_socket [Tracepoint event] syscalls:sys_enter_socketpair [Tracepoint event] syscalls:sys_enter_splice [Tracepoint event] syscalls:sys_enter_statfs [Tracepoint event] syscalls:sys_enter_swapoff [Tracepoint event] syscalls:sys_enter_swapon [Tracepoint event] syscalls:sys_enter_symlink [Tracepoint event] syscalls:sys_enter_symlinkat [Tracepoint event] syscalls:sys_enter_sync [Tracepoint event] syscalls:sys_enter_sync_file_range [Tracepoint event] syscalls:sys_enter_syncfs [Tracepoint event] syscalls:sys_enter_sysctl [Tracepoint event] syscalls:sys_enter_sysfs [Tracepoint event] syscalls:sys_enter_sysinfo [Tracepoint event] syscalls:sys_enter_syslog [Tracepoint event] syscalls:sys_enter_tee [Tracepoint event] syscalls:sys_enter_tgkill [Tracepoint event] syscalls:sys_enter_time [Tracepoint event] syscalls:sys_enter_timer_create [Tracepoint event] syscalls:sys_enter_timer_delete [Tracepoint event] syscalls:sys_enter_timer_getoverrun [Tracepoint event] syscalls:sys_enter_timer_gettime [Tracepoint event] syscalls:sys_enter_timer_settime [Tracepoint event] syscalls:sys_enter_timerfd_create [Tracepoint event] syscalls:sys_enter_timerfd_gettime [Tracepoint event] syscalls:sys_enter_timerfd_settime [Tracepoint event] syscalls:sys_enter_times [Tracepoint event] syscalls:sys_enter_tkill [Tracepoint event] syscalls:sys_enter_truncate [Tracepoint event] syscalls:sys_enter_umask [Tracepoint event] syscalls:sys_enter_umount [Tracepoint event] syscalls:sys_enter_unlink [Tracepoint event] syscalls:sys_enter_unlinkat [Tracepoint event] syscalls:sys_enter_unshare [Tracepoint event] syscalls:sys_enter_ustat [Tracepoint event] syscalls:sys_enter_utime [Tracepoint event] syscalls:sys_enter_utimensat [Tracepoint event] syscalls:sys_enter_utimes [Tracepoint event] syscalls:sys_enter_vhangup [Tracepoint event] syscalls:sys_enter_vmsplice [Tracepoint event] syscalls:sys_enter_wait4 [Tracepoint event] syscalls:sys_enter_waitid [Tracepoint event] syscalls:sys_enter_write [Tracepoint event] syscalls:sys_enter_writev [Tracepoint event] syscalls:sys_exit_accept [Tracepoint event] syscalls:sys_exit_accept4 [Tracepoint event] syscalls:sys_exit_access [Tracepoint event] syscalls:sys_exit_acct [Tracepoint event] syscalls:sys_exit_add_key [Tracepoint event] syscalls:sys_exit_adjtimex [Tracepoint event] syscalls:sys_exit_alarm [Tracepoint event] syscalls:sys_exit_bind [Tracepoint event] syscalls:sys_exit_brk [Tracepoint event] syscalls:sys_exit_capget [Tracepoint event] syscalls:sys_exit_capset [Tracepoint event] syscalls:sys_exit_chdir [Tracepoint event] syscalls:sys_exit_chmod [Tracepoint event] syscalls:sys_exit_chown [Tracepoint event] syscalls:sys_exit_chroot [Tracepoint event] syscalls:sys_exit_clock_adjtime [Tracepoint event] syscalls:sys_exit_clock_getres [Tracepoint event] syscalls:sys_exit_clock_gettime [Tracepoint event] syscalls:sys_exit_clock_nanosleep [Tracepoint event] syscalls:sys_exit_clock_settime [Tracepoint event] syscalls:sys_exit_close [Tracepoint event] syscalls:sys_exit_connect [Tracepoint event] syscalls:sys_exit_creat [Tracepoint event] syscalls:sys_exit_delete_module [Tracepoint event] syscalls:sys_exit_dup [Tracepoint event] syscalls:sys_exit_dup2 [Tracepoint event] syscalls:sys_exit_dup3 [Tracepoint event] syscalls:sys_exit_epoll_create [Tracepoint event] syscalls:sys_exit_epoll_create1 [Tracepoint event] syscalls:sys_exit_epoll_ctl [Tracepoint event] syscalls:sys_exit_epoll_pwait [Tracepoint event] syscalls:sys_exit_epoll_wait [Tracepoint event] syscalls:sys_exit_eventfd [Tracepoint event] syscalls:sys_exit_eventfd2 [Tracepoint event] syscalls:sys_exit_exit [Tracepoint event] syscalls:sys_exit_exit_group [Tracepoint event] syscalls:sys_exit_faccessat [Tracepoint event] syscalls:sys_exit_fadvise64 [Tracepoint event] syscalls:sys_exit_fallocate [Tracepoint event] syscalls:sys_exit_fanotify_init [Tracepoint event] syscalls:sys_exit_fanotify_mark [Tracepoint event] syscalls:sys_exit_fchdir [Tracepoint event] syscalls:sys_exit_fchmod [Tracepoint event] syscalls:sys_exit_fchmodat [Tracepoint event] syscalls:sys_exit_fchown [Tracepoint event] syscalls:sys_exit_fchownat [Tracepoint event] syscalls:sys_exit_fcntl [Tracepoint event] syscalls:sys_exit_fdatasync [Tracepoint event] syscalls:sys_exit_fgetxattr [Tracepoint event] syscalls:sys_exit_finit_module [Tracepoint event] syscalls:sys_exit_flistxattr [Tracepoint event] syscalls:sys_exit_flock [Tracepoint event] syscalls:sys_exit_fremovexattr [Tracepoint event] syscalls:sys_exit_fsetxattr [Tracepoint event] syscalls:sys_exit_fstatfs [Tracepoint event] syscalls:sys_exit_fsync [Tracepoint event] syscalls:sys_exit_ftruncate [Tracepoint event] syscalls:sys_exit_futex [Tracepoint event] syscalls:sys_exit_futimesat [Tracepoint event] syscalls:sys_exit_get_mempolicy [Tracepoint event] syscalls:sys_exit_get_robust_list [Tracepoint event] syscalls:sys_exit_getcpu [Tracepoint event] syscalls:sys_exit_getcwd [Tracepoint event] syscalls:sys_exit_getdents [Tracepoint event] syscalls:sys_exit_getdents64 [Tracepoint event] syscalls:sys_exit_getegid [Tracepoint event] syscalls:sys_exit_geteuid [Tracepoint event] syscalls:sys_exit_getgid [Tracepoint event] syscalls:sys_exit_getgroups [Tracepoint event] syscalls:sys_exit_getitimer [Tracepoint event] syscalls:sys_exit_getpeername [Tracepoint event] syscalls:sys_exit_getpgid [Tracepoint event] syscalls:sys_exit_getpgrp [Tracepoint event] syscalls:sys_exit_getpid [Tracepoint event] syscalls:sys_exit_getppid [Tracepoint event] syscalls:sys_exit_getpriority [Tracepoint event] syscalls:sys_exit_getrandom [Tracepoint event] syscalls:sys_exit_getresgid [Tracepoint event] syscalls:sys_exit_getresuid [Tracepoint event] syscalls:sys_exit_getrlimit [Tracepoint event] syscalls:sys_exit_getrusage [Tracepoint event] syscalls:sys_exit_getsid [Tracepoint event] syscalls:sys_exit_getsockname [Tracepoint event] syscalls:sys_exit_getsockopt [Tracepoint event] syscalls:sys_exit_gettid [Tracepoint event] syscalls:sys_exit_gettimeofday [Tracepoint event] syscalls:sys_exit_getuid [Tracepoint event] syscalls:sys_exit_getxattr [Tracepoint event] syscalls:sys_exit_init_module [Tracepoint event] syscalls:sys_exit_inotify_add_watch [Tracepoint event] syscalls:sys_exit_inotify_init [Tracepoint event] syscalls:sys_exit_inotify_init1 [Tracepoint event] syscalls:sys_exit_inotify_rm_watch [Tracepoint event] syscalls:sys_exit_io_cancel [Tracepoint event] syscalls:sys_exit_io_destroy [Tracepoint event] syscalls:sys_exit_io_getevents [Tracepoint event] syscalls:sys_exit_io_setup [Tracepoint event] syscalls:sys_exit_io_submit [Tracepoint event] syscalls:sys_exit_ioctl [Tracepoint event] syscalls:sys_exit_iopl [Tracepoint event] syscalls:sys_exit_ioprio_get [Tracepoint event] syscalls:sys_exit_ioprio_set [Tracepoint event] syscalls:sys_exit_kcmp [Tracepoint event] syscalls:sys_exit_kexec_load [Tracepoint event] syscalls:sys_exit_keyctl [Tracepoint event] syscalls:sys_exit_kill [Tracepoint event] syscalls:sys_exit_lchown [Tracepoint event] syscalls:sys_exit_lgetxattr [Tracepoint event] syscalls:sys_exit_link [Tracepoint event] syscalls:sys_exit_linkat [Tracepoint event] syscalls:sys_exit_listen [Tracepoint event] syscalls:sys_exit_listxattr [Tracepoint event] syscalls:sys_exit_llistxattr [Tracepoint event] syscalls:sys_exit_lookup_dcookie [Tracepoint event] syscalls:sys_exit_lremovexattr [Tracepoint event] syscalls:sys_exit_lseek [Tracepoint event] syscalls:sys_exit_lsetxattr [Tracepoint event] syscalls:sys_exit_madvise [Tracepoint event] syscalls:sys_exit_mbind [Tracepoint event] syscalls:sys_exit_memfd_create [Tracepoint event] syscalls:sys_exit_migrate_pages [Tracepoint event] syscalls:sys_exit_mincore [Tracepoint event] syscalls:sys_exit_mkdir [Tracepoint event] syscalls:sys_exit_mkdirat [Tracepoint event] syscalls:sys_exit_mknod [Tracepoint event] syscalls:sys_exit_mknodat [Tracepoint event] syscalls:sys_exit_mlock [Tracepoint event] syscalls:sys_exit_mlockall [Tracepoint event] syscalls:sys_exit_mmap [Tracepoint event] syscalls:sys_exit_mount [Tracepoint event] syscalls:sys_exit_move_pages [Tracepoint event] syscalls:sys_exit_mprotect [Tracepoint event] syscalls:sys_exit_mq_getsetattr [Tracepoint event] syscalls:sys_exit_mq_notify [Tracepoint event] syscalls:sys_exit_mq_open [Tracepoint event] syscalls:sys_exit_mq_timedreceive [Tracepoint event] syscalls:sys_exit_mq_timedsend [Tracepoint event] syscalls:sys_exit_mq_unlink [Tracepoint event] syscalls:sys_exit_mremap [Tracepoint event] syscalls:sys_exit_msgctl [Tracepoint event] syscalls:sys_exit_msgget [Tracepoint event] syscalls:sys_exit_msgrcv [Tracepoint event] syscalls:sys_exit_msgsnd [Tracepoint event] syscalls:sys_exit_msync [Tracepoint event] syscalls:sys_exit_munlock [Tracepoint event] syscalls:sys_exit_munlockall [Tracepoint event] syscalls:sys_exit_munmap [Tracepoint event] syscalls:sys_exit_name_to_handle_at [Tracepoint event] syscalls:sys_exit_nanosleep [Tracepoint event] syscalls:sys_exit_newfstat [Tracepoint event] syscalls:sys_exit_newfstatat [Tracepoint event] syscalls:sys_exit_newlstat [Tracepoint event] syscalls:sys_exit_newstat [Tracepoint event] syscalls:sys_exit_newuname [Tracepoint event] syscalls:sys_exit_open [Tracepoint event] syscalls:sys_exit_open_by_handle_at [Tracepoint event] syscalls:sys_exit_openat [Tracepoint event] syscalls:sys_exit_pause [Tracepoint event] syscalls:sys_exit_perf_event_open [Tracepoint event] syscalls:sys_exit_personality [Tracepoint event] syscalls:sys_exit_pipe [Tracepoint event] syscalls:sys_exit_pipe2 [Tracepoint event] syscalls:sys_exit_pivot_root [Tracepoint event] syscalls:sys_exit_poll [Tracepoint event] syscalls:sys_exit_ppoll [Tracepoint event] syscalls:sys_exit_prctl [Tracepoint event] syscalls:sys_exit_pread64 [Tracepoint event] syscalls:sys_exit_preadv [Tracepoint event] syscalls:sys_exit_prlimit64 [Tracepoint event] syscalls:sys_exit_process_vm_readv [Tracepoint event] syscalls:sys_exit_process_vm_writev [Tracepoint event] syscalls:sys_exit_pselect6 [Tracepoint event] syscalls:sys_exit_ptrace [Tracepoint event] syscalls:sys_exit_pwrite64 [Tracepoint event] syscalls:sys_exit_pwritev [Tracepoint event] syscalls:sys_exit_quotactl [Tracepoint event] syscalls:sys_exit_read [Tracepoint event] syscalls:sys_exit_readahead [Tracepoint event] syscalls:sys_exit_readlink [Tracepoint event] syscalls:sys_exit_readlinkat [Tracepoint event] syscalls:sys_exit_readv [Tracepoint event] syscalls:sys_exit_reboot [Tracepoint event] syscalls:sys_exit_recvfrom [Tracepoint event] syscalls:sys_exit_recvmmsg [Tracepoint event] syscalls:sys_exit_recvmsg [Tracepoint event] syscalls:sys_exit_remap_file_pages [Tracepoint event] syscalls:sys_exit_removexattr [Tracepoint event] syscalls:sys_exit_rename [Tracepoint event] syscalls:sys_exit_renameat [Tracepoint event] syscalls:sys_exit_renameat2 [Tracepoint event] syscalls:sys_exit_request_key [Tracepoint event] syscalls:sys_exit_restart_syscall [Tracepoint event] syscalls:sys_exit_rmdir [Tracepoint event] syscalls:sys_exit_rt_sigaction [Tracepoint event] syscalls:sys_exit_rt_sigpending [Tracepoint event] syscalls:sys_exit_rt_sigprocmask [Tracepoint event] syscalls:sys_exit_rt_sigqueueinfo [Tracepoint event] syscalls:sys_exit_rt_sigsuspend [Tracepoint event] syscalls:sys_exit_rt_sigtimedwait [Tracepoint event] syscalls:sys_exit_rt_tgsigqueueinfo [Tracepoint event] syscalls:sys_exit_sched_get_priority_max [Tracepoint event] syscalls:sys_exit_sched_get_priority_min [Tracepoint event] syscalls:sys_exit_sched_getaffinity [Tracepoint event] syscalls:sys_exit_sched_getattr [Tracepoint event] syscalls:sys_exit_sched_getparam [Tracepoint event] syscalls:sys_exit_sched_getscheduler [Tracepoint event] syscalls:sys_exit_sched_rr_get_interval [Tracepoint event] syscalls:sys_exit_sched_setaffinity [Tracepoint event] syscalls:sys_exit_sched_setattr [Tracepoint event] syscalls:sys_exit_sched_setparam [Tracepoint event] syscalls:sys_exit_sched_setscheduler [Tracepoint event] syscalls:sys_exit_sched_yield [Tracepoint event] syscalls:sys_exit_seccomp [Tracepoint event] syscalls:sys_exit_select [Tracepoint event] syscalls:sys_exit_semctl [Tracepoint event] syscalls:sys_exit_semget [Tracepoint event] syscalls:sys_exit_semop [Tracepoint event] syscalls:sys_exit_semtimedop [Tracepoint event] syscalls:sys_exit_sendfile64 [Tracepoint event] syscalls:sys_exit_sendmmsg [Tracepoint event] syscalls:sys_exit_sendmsg [Tracepoint event] syscalls:sys_exit_sendto [Tracepoint event] syscalls:sys_exit_set_mempolicy [Tracepoint event] syscalls:sys_exit_set_robust_list [Tracepoint event] syscalls:sys_exit_set_tid_address [Tracepoint event] syscalls:sys_exit_setdomainname [Tracepoint event] syscalls:sys_exit_setfsgid [Tracepoint event] syscalls:sys_exit_setfsuid [Tracepoint event] syscalls:sys_exit_setgid [Tracepoint event] syscalls:sys_exit_setgroups [Tracepoint event] syscalls:sys_exit_sethostname [Tracepoint event] syscalls:sys_exit_setitimer [Tracepoint event] syscalls:sys_exit_setns [Tracepoint event] syscalls:sys_exit_setpgid [Tracepoint event] syscalls:sys_exit_setpriority [Tracepoint event] syscalls:sys_exit_setregid [Tracepoint event] syscalls:sys_exit_setresgid [Tracepoint event] syscalls:sys_exit_setresuid [Tracepoint event] syscalls:sys_exit_setreuid [Tracepoint event] syscalls:sys_exit_setrlimit [Tracepoint event] syscalls:sys_exit_setsid [Tracepoint event] syscalls:sys_exit_setsockopt [Tracepoint event] syscalls:sys_exit_settimeofday [Tracepoint event] syscalls:sys_exit_setuid [Tracepoint event] syscalls:sys_exit_setxattr [Tracepoint event] syscalls:sys_exit_shmat [Tracepoint event] syscalls:sys_exit_shmctl [Tracepoint event] syscalls:sys_exit_shmdt [Tracepoint event] syscalls:sys_exit_shmget [Tracepoint event] syscalls:sys_exit_shutdown [Tracepoint event] syscalls:sys_exit_sigaltstack [Tracepoint event] syscalls:sys_exit_signalfd [Tracepoint event] syscalls:sys_exit_signalfd4 [Tracepoint event] syscalls:sys_exit_socket [Tracepoint event] syscalls:sys_exit_socketpair [Tracepoint event] syscalls:sys_exit_splice [Tracepoint event] syscalls:sys_exit_statfs [Tracepoint event] syscalls:sys_exit_swapoff [Tracepoint event] syscalls:sys_exit_swapon [Tracepoint event] syscalls:sys_exit_symlink [Tracepoint event] syscalls:sys_exit_symlinkat [Tracepoint event] syscalls:sys_exit_sync [Tracepoint event] syscalls:sys_exit_sync_file_range [Tracepoint event] syscalls:sys_exit_syncfs [Tracepoint event] syscalls:sys_exit_sysctl [Tracepoint event] syscalls:sys_exit_sysfs [Tracepoint event] syscalls:sys_exit_sysinfo [Tracepoint event] syscalls:sys_exit_syslog [Tracepoint event] syscalls:sys_exit_tee [Tracepoint event] syscalls:sys_exit_tgkill [Tracepoint event] syscalls:sys_exit_time [Tracepoint event] syscalls:sys_exit_timer_create [Tracepoint event] syscalls:sys_exit_timer_delete [Tracepoint event] syscalls:sys_exit_timer_getoverrun [Tracepoint event] syscalls:sys_exit_timer_gettime [Tracepoint event] syscalls:sys_exit_timer_settime [Tracepoint event] syscalls:sys_exit_timerfd_create [Tracepoint event] syscalls:sys_exit_timerfd_gettime [Tracepoint event] syscalls:sys_exit_timerfd_settime [Tracepoint event] syscalls:sys_exit_times [Tracepoint event] syscalls:sys_exit_tkill [Tracepoint event] syscalls:sys_exit_truncate [Tracepoint event] syscalls:sys_exit_umask [Tracepoint event] syscalls:sys_exit_umount [Tracepoint event] syscalls:sys_exit_unlink [Tracepoint event] syscalls:sys_exit_unlinkat [Tracepoint event] syscalls:sys_exit_unshare [Tracepoint event] syscalls:sys_exit_ustat [Tracepoint event] syscalls:sys_exit_utime [Tracepoint event] syscalls:sys_exit_utimensat [Tracepoint event] syscalls:sys_exit_utimes [Tracepoint event] syscalls:sys_exit_vhangup [Tracepoint event] syscalls:sys_exit_vmsplice [Tracepoint event] syscalls:sys_exit_wait4 [Tracepoint event] syscalls:sys_exit_waitid [Tracepoint event] syscalls:sys_exit_write [Tracepoint event] syscalls:sys_exit_writev [Tracepoint event] task:task_newtask [Tracepoint event] task:task_rename [Tracepoint event] thermal:cdev_update [Tracepoint event] thermal:thermal_power_cpu_get_power [Tracepoint event] thermal:thermal_power_cpu_limit [Tracepoint event] thermal:thermal_temperature [Tracepoint event] thermal:thermal_zone_trip [Tracepoint event] timer:hrtimer_cancel [Tracepoint event] timer:hrtimer_expire_entry [Tracepoint event] timer:hrtimer_expire_exit [Tracepoint event] timer:hrtimer_init [Tracepoint event] timer:hrtimer_start [Tracepoint event] timer:itimer_expire [Tracepoint event] timer:itimer_state [Tracepoint event] timer:tick_stop [Tracepoint event] timer:timer_cancel [Tracepoint event] timer:timer_expire_entry [Tracepoint event] timer:timer_expire_exit [Tracepoint event] timer:timer_init [Tracepoint event] timer:timer_start [Tracepoint event] tlb:tlb_flush [Tracepoint event] udp:udp_fail_queue_rcv_skb [Tracepoint event] vmscan:mm_shrink_slab_end [Tracepoint event] vmscan:mm_shrink_slab_start [Tracepoint event] vmscan:mm_vmscan_direct_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_direct_reclaim_end [Tracepoint event] vmscan:mm_vmscan_kswapd_sleep [Tracepoint event] vmscan:mm_vmscan_kswapd_wake [Tracepoint event] vmscan:mm_vmscan_lru_isolate [Tracepoint event] vmscan:mm_vmscan_lru_shrink_inactive [Tracepoint event] vmscan:mm_vmscan_memcg_isolate [Tracepoint event] vmscan:mm_vmscan_memcg_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_memcg_reclaim_end [Tracepoint event] vmscan:mm_vmscan_memcg_softlimit_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_memcg_softlimit_reclaim_end [Tracepoint event] vmscan:mm_vmscan_wakeup_kswapd [Tracepoint event] vmscan:mm_vmscan_writepage [Tracepoint event] vsyscall:emulate_vsyscall [Tracepoint event] workqueue:workqueue_activate_work [Tracepoint event] workqueue:workqueue_execute_end [Tracepoint event] workqueue:workqueue_execute_start [Tracepoint event] workqueue:workqueue_queue_work [Tracepoint event] writeback:balance_dirty_pages [Tracepoint event] writeback:bdi_dirty_ratelimit [Tracepoint event] writeback:global_dirty_state [Tracepoint event] writeback:wbc_writepage [Tracepoint event] writeback:writeback_bdi_register [Tracepoint event] writeback:writeback_congestion_wait [Tracepoint event] writeback:writeback_dirty_inode [Tracepoint event] writeback:writeback_dirty_inode_enqueue [Tracepoint event] writeback:writeback_dirty_inode_start [Tracepoint event] writeback:writeback_dirty_page [Tracepoint event] writeback:writeback_exec [Tracepoint event] writeback:writeback_lazytime [Tracepoint event] writeback:writeback_lazytime_iput [Tracepoint event] writeback:writeback_mark_inode_dirty [Tracepoint event] writeback:writeback_nowork [Tracepoint event] writeback:writeback_pages_written [Tracepoint event] writeback:writeback_queue [Tracepoint event] writeback:writeback_queue_io [Tracepoint event] writeback:writeback_sb_inodes_requeue [Tracepoint event] writeback:writeback_single_inode [Tracepoint event] writeback:writeback_single_inode_start [Tracepoint event] writeback:writeback_start [Tracepoint event] writeback:writeback_wait [Tracepoint event] writeback:writeback_wait_iff_congested [Tracepoint event] writeback:writeback_wake_background [Tracepoint event] writeback:writeback_write_inode [Tracepoint event] writeback:writeback_write_inode_start [Tracepoint event] writeback:writeback_written [Tracepoint event] xen:xen_cpu_load_idt [Tracepoint event] xen:xen_cpu_set_ldt [Tracepoint event] xen:xen_cpu_write_gdt_entry [Tracepoint event] xen:xen_cpu_write_idt_entry [Tracepoint event] xen:xen_cpu_write_ldt_entry [Tracepoint event] xen:xen_mc_batch [Tracepoint event] xen:xen_mc_callback [Tracepoint event] xen:xen_mc_entry [Tracepoint event] xen:xen_mc_entry_alloc [Tracepoint event] xen:xen_mc_extend_args [Tracepoint event] xen:xen_mc_flush [Tracepoint event] xen:xen_mc_flush_reason [Tracepoint event] xen:xen_mc_issue [Tracepoint event] xen:xen_mmu_alloc_ptpage [Tracepoint event] xen:xen_mmu_flush_tlb [Tracepoint event] xen:xen_mmu_flush_tlb_all [Tracepoint event] xen:xen_mmu_flush_tlb_others [Tracepoint event] xen:xen_mmu_flush_tlb_single [Tracepoint event] xen:xen_mmu_pgd_clear [Tracepoint event] xen:xen_mmu_pgd_pin [Tracepoint event] xen:xen_mmu_pgd_unpin [Tracepoint event] xen:xen_mmu_pmd_clear [Tracepoint event] xen:xen_mmu_pte_clear [Tracepoint event] xen:xen_mmu_ptep_modify_prot_commit [Tracepoint event] xen:xen_mmu_ptep_modify_prot_start [Tracepoint event] xen:xen_mmu_pud_clear [Tracepoint event] xen:xen_mmu_release_ptpage [Tracepoint event] xen:xen_mmu_set_domain_pte [Tracepoint event] xen:xen_mmu_set_pgd [Tracepoint event] xen:xen_mmu_set_pmd [Tracepoint event] xen:xen_mmu_set_pte [Tracepoint event] xen:xen_mmu_set_pte_at [Tracepoint event] xen:xen_mmu_set_pte_atomic [Tracepoint event] xen:xen_mmu_set_pud [Tracepoint event] xen:xen_mmu_write_cr3 [Tracepoint event] xfs:xfs_agf [Tracepoint event] xfs:xfs_ail_delete [Tracepoint event] xfs:xfs_ail_flushing [Tracepoint event] xfs:xfs_ail_insert [Tracepoint event] xfs:xfs_ail_locked [Tracepoint event] xfs:xfs_ail_move [Tracepoint event] xfs:xfs_ail_pinned [Tracepoint event] xfs:xfs_ail_push [Tracepoint event] xfs:xfs_alloc_exact_done [Tracepoint event] xfs:xfs_alloc_exact_error [Tracepoint event] xfs:xfs_alloc_exact_notfound [Tracepoint event] xfs:xfs_alloc_file_space [Tracepoint event] xfs:xfs_alloc_near_busy [Tracepoint event] xfs:xfs_alloc_near_error [Tracepoint event] xfs:xfs_alloc_near_first [Tracepoint event] xfs:xfs_alloc_near_greater [Tracepoint event] xfs:xfs_alloc_near_lesser [Tracepoint event] xfs:xfs_alloc_near_noentry [Tracepoint event] xfs:xfs_alloc_near_nominleft [Tracepoint event] xfs:xfs_alloc_read_agf [Tracepoint event] xfs:xfs_alloc_size_busy [Tracepoint event] xfs:xfs_alloc_size_done [Tracepoint event] xfs:xfs_alloc_size_error [Tracepoint event] xfs:xfs_alloc_size_neither [Tracepoint event] xfs:xfs_alloc_size_noentry [Tracepoint event] xfs:xfs_alloc_size_nominleft [Tracepoint event] xfs:xfs_alloc_small_done [Tracepoint event] xfs:xfs_alloc_small_error [Tracepoint event] xfs:xfs_alloc_small_freelist [Tracepoint event] xfs:xfs_alloc_small_notenough [Tracepoint event] xfs:xfs_alloc_vextent_allfailed [Tracepoint event] xfs:xfs_alloc_vextent_badargs [Tracepoint event] xfs:xfs_alloc_vextent_loopfailed [Tracepoint event] xfs:xfs_alloc_vextent_noagbp [Tracepoint event] xfs:xfs_alloc_vextent_nofix [Tracepoint event] xfs:xfs_attr_fillstate [Tracepoint event] xfs:xfs_attr_leaf_add [Tracepoint event] xfs:xfs_attr_leaf_add_new [Tracepoint event] xfs:xfs_attr_leaf_add_old [Tracepoint event] xfs:xfs_attr_leaf_add_work [Tracepoint event] xfs:xfs_attr_leaf_addname [Tracepoint event] xfs:xfs_attr_leaf_clearflag [Tracepoint event] xfs:xfs_attr_leaf_compact [Tracepoint event] xfs:xfs_attr_leaf_create [Tracepoint event] xfs:xfs_attr_leaf_flipflags [Tracepoint event] xfs:xfs_attr_leaf_get [Tracepoint event] xfs:xfs_attr_leaf_list [Tracepoint event] xfs:xfs_attr_leaf_lookup [Tracepoint event] xfs:xfs_attr_leaf_rebalance [Tracepoint event] xfs:xfs_attr_leaf_remove [Tracepoint event] xfs:xfs_attr_leaf_removename [Tracepoint event] xfs:xfs_attr_leaf_replace [Tracepoint event] xfs:xfs_attr_leaf_setflag [Tracepoint event] xfs:xfs_attr_leaf_split [Tracepoint event] xfs:xfs_attr_leaf_split_after [Tracepoint event] xfs:xfs_attr_leaf_split_before [Tracepoint event] xfs:xfs_attr_leaf_to_node [Tracepoint event] xfs:xfs_attr_leaf_to_sf [Tracepoint event] xfs:xfs_attr_leaf_toosmall [Tracepoint event] xfs:xfs_attr_leaf_unbalance [Tracepoint event] xfs:xfs_attr_list_add [Tracepoint event] xfs:xfs_attr_list_full [Tracepoint event] xfs:xfs_attr_list_leaf [Tracepoint event] xfs:xfs_attr_list_leaf_end [Tracepoint event] xfs:xfs_attr_list_node_descend [Tracepoint event] xfs:xfs_attr_list_notfound [Tracepoint event] xfs:xfs_attr_list_sf [Tracepoint event] xfs:xfs_attr_list_sf_all [Tracepoint event] xfs:xfs_attr_list_wrong_blk [Tracepoint event] xfs:xfs_attr_node_addname [Tracepoint event] xfs:xfs_attr_node_get [Tracepoint event] xfs:xfs_attr_node_list [Tracepoint event] xfs:xfs_attr_node_lookup [Tracepoint event] xfs:xfs_attr_node_removename [Tracepoint event] xfs:xfs_attr_node_replace [Tracepoint event] xfs:xfs_attr_refillstate [Tracepoint event] xfs:xfs_attr_rmtval_get [Tracepoint event] xfs:xfs_attr_rmtval_remove [Tracepoint event] xfs:xfs_attr_rmtval_set [Tracepoint event] xfs:xfs_attr_sf_add [Tracepoint event] xfs:xfs_attr_sf_addname [Tracepoint event] xfs:xfs_attr_sf_create [Tracepoint event] xfs:xfs_attr_sf_lookup [Tracepoint event] xfs:xfs_attr_sf_remove [Tracepoint event] xfs:xfs_attr_sf_removename [Tracepoint event] xfs:xfs_attr_sf_to_leaf [Tracepoint event] xfs:xfs_bdstrat_shut [Tracepoint event] xfs:xfs_bmap_post_update [Tracepoint event] xfs:xfs_bmap_pre_update [Tracepoint event] xfs:xfs_btree_corrupt [Tracepoint event] xfs:xfs_buf_bawrite [Tracepoint event] xfs:xfs_buf_delwri_queue [Tracepoint event] xfs:xfs_buf_delwri_queued [Tracepoint event] xfs:xfs_buf_delwri_split [Tracepoint event] xfs:xfs_buf_error_relse [Tracepoint event] xfs:xfs_buf_find [Tracepoint event] xfs:xfs_buf_free [Tracepoint event] xfs:xfs_buf_get [Tracepoint event] xfs:xfs_buf_get_uncached [Tracepoint event] xfs:xfs_buf_hold [Tracepoint event] xfs:xfs_buf_init [Tracepoint event] xfs:xfs_buf_iodone [Tracepoint event] xfs:xfs_buf_ioerror [Tracepoint event] xfs:xfs_buf_iowait [Tracepoint event] xfs:xfs_buf_iowait_done [Tracepoint event] xfs:xfs_buf_item_committed [Tracepoint event] xfs:xfs_buf_item_format [Tracepoint event] xfs:xfs_buf_item_format_ordered [Tracepoint event] xfs:xfs_buf_item_format_stale [Tracepoint event] xfs:xfs_buf_item_iodone [Tracepoint event] xfs:xfs_buf_item_iodone_async [Tracepoint event] xfs:xfs_buf_item_ordered [Tracepoint event] xfs:xfs_buf_item_pin [Tracepoint event] xfs:xfs_buf_item_push [Tracepoint event] xfs:xfs_buf_item_relse [Tracepoint event] xfs:xfs_buf_item_size [Tracepoint event] xfs:xfs_buf_item_size_ordered [Tracepoint event] xfs:xfs_buf_item_size_stale [Tracepoint event] xfs:xfs_buf_item_unlock [Tracepoint event] xfs:xfs_buf_item_unlock_stale [Tracepoint event] xfs:xfs_buf_item_unpin [Tracepoint event] xfs:xfs_buf_item_unpin_stale [Tracepoint event] xfs:xfs_buf_lock [Tracepoint event] xfs:xfs_buf_lock_done [Tracepoint event] xfs:xfs_buf_read [Tracepoint event] xfs:xfs_buf_rele [Tracepoint event] xfs:xfs_buf_submit [Tracepoint event] xfs:xfs_buf_submit_wait [Tracepoint event] xfs:xfs_buf_trylock [Tracepoint event] xfs:xfs_buf_unlock [Tracepoint event] xfs:xfs_buf_wait_buftarg [Tracepoint event] xfs:xfs_bunmap [Tracepoint event] xfs:xfs_collapse_file_space [Tracepoint event] xfs:xfs_create [Tracepoint event] xfs:xfs_da_btree_corrupt [Tracepoint event] xfs:xfs_da_fixhashpath [Tracepoint event] xfs:xfs_da_grow_inode [Tracepoint event] xfs:xfs_da_join [Tracepoint event] xfs:xfs_da_link_after [Tracepoint event] xfs:xfs_da_link_before [Tracepoint event] xfs:xfs_da_node_add [Tracepoint event] xfs:xfs_da_node_create [Tracepoint event] xfs:xfs_da_node_rebalance [Tracepoint event] xfs:xfs_da_node_remove [Tracepoint event] xfs:xfs_da_node_split [Tracepoint event] xfs:xfs_da_node_toosmall [Tracepoint event] xfs:xfs_da_node_unbalance [Tracepoint event] xfs:xfs_da_path_shift [Tracepoint event] xfs:xfs_da_root_join [Tracepoint event] xfs:xfs_da_root_split [Tracepoint event] xfs:xfs_da_shrink_inode [Tracepoint event] xfs:xfs_da_split [Tracepoint event] xfs:xfs_da_swap_lastblock [Tracepoint event] xfs:xfs_da_unlink_back [Tracepoint event] xfs:xfs_da_unlink_forward [Tracepoint event] xfs:xfs_delalloc_enospc [Tracepoint event] xfs:xfs_destroy_inode [Tracepoint event] xfs:xfs_dir2_block_addname [Tracepoint event] xfs:xfs_dir2_block_lookup [Tracepoint event] xfs:xfs_dir2_block_removename [Tracepoint event] xfs:xfs_dir2_block_replace [Tracepoint event] xfs:xfs_dir2_block_to_leaf [Tracepoint event] xfs:xfs_dir2_block_to_sf [Tracepoint event] xfs:xfs_dir2_grow_inode [Tracepoint event] xfs:xfs_dir2_leaf_addname [Tracepoint event] xfs:xfs_dir2_leaf_lookup [Tracepoint event] xfs:xfs_dir2_leaf_removename [Tracepoint event] xfs:xfs_dir2_leaf_replace [Tracepoint event] xfs:xfs_dir2_leaf_to_block [Tracepoint event] xfs:xfs_dir2_leaf_to_node [Tracepoint event] xfs:xfs_dir2_leafn_add [Tracepoint event] xfs:xfs_dir2_leafn_moveents [Tracepoint event] xfs:xfs_dir2_leafn_remove [Tracepoint event] xfs:xfs_dir2_node_addname [Tracepoint event] xfs:xfs_dir2_node_lookup [Tracepoint event] xfs:xfs_dir2_node_removename [Tracepoint event] xfs:xfs_dir2_node_replace [Tracepoint event] xfs:xfs_dir2_node_to_leaf [Tracepoint event] xfs:xfs_dir2_sf_addname [Tracepoint event] xfs:xfs_dir2_sf_create [Tracepoint event] xfs:xfs_dir2_sf_lookup [Tracepoint event] xfs:xfs_dir2_sf_removename [Tracepoint event] xfs:xfs_dir2_sf_replace [Tracepoint event] xfs:xfs_dir2_sf_to_block [Tracepoint event] xfs:xfs_dir2_sf_toino4 [Tracepoint event] xfs:xfs_dir2_sf_toino8 [Tracepoint event] xfs:xfs_dir2_shrink_inode [Tracepoint event] xfs:xfs_dir_fsync [Tracepoint event] xfs:xfs_discard_busy [Tracepoint event] xfs:xfs_discard_exclude [Tracepoint event] xfs:xfs_discard_extent [Tracepoint event] xfs:xfs_discard_toosmall [Tracepoint event] xfs:xfs_dqadjust [Tracepoint event] xfs:xfs_dqalloc [Tracepoint event] xfs:xfs_dqattach_found [Tracepoint event] xfs:xfs_dqattach_get [Tracepoint event] xfs:xfs_dqflush [Tracepoint event] xfs:xfs_dqflush_done [Tracepoint event] xfs:xfs_dqflush_force [Tracepoint event] xfs:xfs_dqget_dup [Tracepoint event] xfs:xfs_dqget_freeing [Tracepoint event] xfs:xfs_dqget_hit [Tracepoint event] xfs:xfs_dqget_miss [Tracepoint event] xfs:xfs_dqput [Tracepoint event] xfs:xfs_dqput_free [Tracepoint event] xfs:xfs_dqput_wait [Tracepoint event] xfs:xfs_dqread [Tracepoint event] xfs:xfs_dqread_fail [Tracepoint event] xfs:xfs_dqreclaim_busy [Tracepoint event] xfs:xfs_dqreclaim_dirty [Tracepoint event] xfs:xfs_dqreclaim_done [Tracepoint event] xfs:xfs_dqreclaim_want [Tracepoint event] xfs:xfs_dqrele [Tracepoint event] xfs:xfs_dqtobp_read [Tracepoint event] xfs:xfs_dquot_dqalloc [Tracepoint event] xfs:xfs_dquot_dqdetach [Tracepoint event] xfs:xfs_evict_inode [Tracepoint event] xfs:xfs_extent_busy [Tracepoint event] xfs:xfs_extent_busy_clear [Tracepoint event] xfs:xfs_extent_busy_enomem [Tracepoint event] xfs:xfs_extent_busy_force [Tracepoint event] xfs:xfs_extent_busy_reuse [Tracepoint event] xfs:xfs_extent_busy_trim [Tracepoint event] xfs:xfs_extlist [Tracepoint event] xfs:xfs_file_buffered_write [Tracepoint event] xfs:xfs_file_compat_ioctl [Tracepoint event] xfs:xfs_file_direct_write [Tracepoint event] xfs:xfs_file_fsync [Tracepoint event] xfs:xfs_file_ioctl [Tracepoint event] xfs:xfs_file_read [Tracepoint event] xfs:xfs_file_splice_read [Tracepoint event] xfs:xfs_filemap_fault [Tracepoint event] xfs:xfs_filemap_page_mkwrite [Tracepoint event] xfs:xfs_filestream_free [Tracepoint event] xfs:xfs_filestream_lookup [Tracepoint event] xfs:xfs_filestream_pick [Tracepoint event] xfs:xfs_filestream_scan [Tracepoint event] xfs:xfs_free_extent [Tracepoint event] xfs:xfs_free_file_space [Tracepoint event] xfs:xfs_gbmap_direct [Tracepoint event] xfs:xfs_gbmap_direct_endio [Tracepoint event] xfs:xfs_gbmap_direct_new [Tracepoint event] xfs:xfs_gbmap_direct_none [Tracepoint event] xfs:xfs_gbmap_direct_update [Tracepoint event] xfs:xfs_get_acl [Tracepoint event] xfs:xfs_get_blocks_alloc [Tracepoint event] xfs:xfs_get_blocks_found [Tracepoint event] xfs:xfs_get_blocks_notfound [Tracepoint event] xfs:xfs_getattr [Tracepoint event] xfs:xfs_ialloc_read_agi [Tracepoint event] xfs:xfs_iext_insert [Tracepoint event] xfs:xfs_iext_remove [Tracepoint event] xfs:xfs_iget_hit [Tracepoint event] xfs:xfs_iget_miss [Tracepoint event] xfs:xfs_iget_reclaim [Tracepoint event] xfs:xfs_iget_reclaim_fail [Tracepoint event] xfs:xfs_iget_skip [Tracepoint event] xfs:xfs_ihold [Tracepoint event] xfs:xfs_ilock [Tracepoint event] xfs:xfs_ilock_demote [Tracepoint event] xfs:xfs_ilock_nowait [Tracepoint event] xfs:xfs_inactive_symlink [Tracepoint event] xfs:xfs_inode_clear_eofblocks_tag [Tracepoint event] xfs:xfs_inode_free_eofblocks_invalid [Tracepoint event] xfs:xfs_inode_item_push [Tracepoint event] xfs:xfs_inode_pin [Tracepoint event] xfs:xfs_inode_set_eofblocks_tag [Tracepoint event] xfs:xfs_inode_unpin [Tracepoint event] xfs:xfs_inode_unpin_nowait [Tracepoint event] xfs:xfs_insert_file_space [Tracepoint event] xfs:xfs_invalidatepage [Tracepoint event] xfs:xfs_ioctl_setattr [Tracepoint event] xfs:xfs_iomap_prealloc_size [Tracepoint event] xfs:xfs_irec_merge_post [Tracepoint event] xfs:xfs_irec_merge_pre [Tracepoint event] xfs:xfs_irele [Tracepoint event] xfs:xfs_itruncate_extents_end [Tracepoint event] xfs:xfs_itruncate_extents_start [Tracepoint event] xfs:xfs_iunlock [Tracepoint event] xfs:xfs_link [Tracepoint event] xfs:xfs_log_assign_tail_lsn [Tracepoint event] xfs:xfs_log_done_nonperm [Tracepoint event] xfs:xfs_log_done_perm [Tracepoint event] xfs:xfs_log_force [Tracepoint event] xfs:xfs_log_grant_sleep [Tracepoint event] xfs:xfs_log_grant_wake [Tracepoint event] xfs:xfs_log_grant_wake_up [Tracepoint event] xfs:xfs_log_recover_buf_cancel [Tracepoint event] xfs:xfs_log_recover_buf_cancel_add [Tracepoint event] xfs:xfs_log_recover_buf_cancel_ref_inc [Tracepoint event] xfs:xfs_log_recover_buf_dquot_buf [Tracepoint event] xfs:xfs_log_recover_buf_inode_buf [Tracepoint event] xfs:xfs_log_recover_buf_not_cancel [Tracepoint event] xfs:xfs_log_recover_buf_recover [Tracepoint event] xfs:xfs_log_recover_buf_reg_buf [Tracepoint event] xfs:xfs_log_recover_inode_cancel [Tracepoint event] xfs:xfs_log_recover_inode_recover [Tracepoint event] xfs:xfs_log_recover_inode_skip [Tracepoint event] xfs:xfs_log_recover_item_add [Tracepoint event] xfs:xfs_log_recover_item_add_cont [Tracepoint event] xfs:xfs_log_recover_item_recover [Tracepoint event] xfs:xfs_log_recover_item_reorder_head [Tracepoint event] xfs:xfs_log_recover_item_reorder_tail [Tracepoint event] xfs:xfs_log_regrant [Tracepoint event] xfs:xfs_log_regrant_exit [Tracepoint event] xfs:xfs_log_regrant_reserve_enter [Tracepoint event] xfs:xfs_log_regrant_reserve_exit [Tracepoint event] xfs:xfs_log_regrant_reserve_sub [Tracepoint event] xfs:xfs_log_reserve [Tracepoint event] xfs:xfs_log_reserve_exit [Tracepoint event] xfs:xfs_log_umount_write [Tracepoint event] xfs:xfs_log_ungrant_enter [Tracepoint event] xfs:xfs_log_ungrant_exit [Tracepoint event] xfs:xfs_log_ungrant_sub [Tracepoint event] xfs:xfs_lookup [Tracepoint event] xfs:xfs_map_blocks_alloc [Tracepoint event] xfs:xfs_map_blocks_found [Tracepoint event] xfs:xfs_pagecache_inval [Tracepoint event] xfs:xfs_perag_clear_eofblocks [Tracepoint event] xfs:xfs_perag_clear_reclaim [Tracepoint event] xfs:xfs_perag_get [Tracepoint event] xfs:xfs_perag_get_tag [Tracepoint event] xfs:xfs_perag_put [Tracepoint event] xfs:xfs_perag_set_eofblocks [Tracepoint event] xfs:xfs_perag_set_reclaim [Tracepoint event] xfs:xfs_read_agf [Tracepoint event] xfs:xfs_read_agi [Tracepoint event] xfs:xfs_readdir [Tracepoint event] xfs:xfs_readlink [Tracepoint event] xfs:xfs_releasepage [Tracepoint event] xfs:xfs_remove [Tracepoint event] xfs:xfs_rename [Tracepoint event] xfs:xfs_reset_dqcounts [Tracepoint event] xfs:xfs_setattr [Tracepoint event] xfs:xfs_setfilesize [Tracepoint event] xfs:xfs_swap_extent_after [Tracepoint event] xfs:xfs_swap_extent_before [Tracepoint event] xfs:xfs_symlink [Tracepoint event] xfs:xfs_trans_bhold [Tracepoint event] xfs:xfs_trans_bhold_release [Tracepoint event] xfs:xfs_trans_binval [Tracepoint event] xfs:xfs_trans_bjoin [Tracepoint event] xfs:xfs_trans_brelse [Tracepoint event] xfs:xfs_trans_buf_ordered [Tracepoint event] xfs:xfs_trans_commit_lsn [Tracepoint event] xfs:xfs_trans_get_buf [Tracepoint event] xfs:xfs_trans_get_buf_recur [Tracepoint event] xfs:xfs_trans_getsb [Tracepoint event] xfs:xfs_trans_getsb_recur [Tracepoint event] xfs:xfs_trans_log_buf [Tracepoint event] xfs:xfs_trans_read_buf [Tracepoint event] xfs:xfs_trans_read_buf_io [Tracepoint event] xfs:xfs_trans_read_buf_recur [Tracepoint event] xfs:xfs_trans_read_buf_shut [Tracepoint event] xfs:xfs_unwritten_convert [Tracepoint event] xfs:xfs_update_time [Tracepoint event] xfs:xfs_vm_bmap [Tracepoint event] xfs:xfs_writepage [Tracepoint event] xfs:xfs_zero_file_space [Tracepoint event] xhci-hcd:xhci_address_ctx [Tracepoint event] xhci-hcd:xhci_cmd_completion [Tracepoint event] xhci-hcd:xhci_dbg_address [Tracepoint event] xhci-hcd:xhci_dbg_cancel_urb [Tracepoint event] xhci-hcd:xhci_dbg_context_change [Tracepoint event] xhci-hcd:xhci_dbg_init [Tracepoint event] xhci-hcd:xhci_dbg_quirks [Tracepoint event] xhci-hcd:xhci_dbg_reset_ep [Tracepoint event] xhci-hcd:xhci_dbg_ring_expansion [Tracepoint event]

[rocallahan]

For a performance event to be usable for rr, the event must fire at least once on every iteration of a loop, and every time you execute a given sequence of instructions in user space, the same number of events must be counted. So events that unpredictably fire an extra time (e.g. because of speculative state, or a hardware interrupt, or a page fault) are useless for rr. We know that on every Intel family none of events listed above are adequate, and my guess is that that will be true for AMD too, but I could be wrong.

If you want to work on this, I would start with the retired-branches event. Set up a perf session to count retired-branches and use strace to figure out the magic numbers to pass to perf_event_open. Once you have that, try plugging it into rr in PerfCounters.cc and running the rr tests --- I'd start with the alarm test since that's about the simplest test for performance counter consistency. If you can get that test to pass reliably then there may be some chance.

[edwintorok]

Thanks for the hint, I've made a little progress. I was experimenting with perf stat -e instructions:u,r5301cf:u ./a.out first (on that test program with 1 million instructions), and disabling cpuidle C2 made it somewhat more consistent, so:

cpupower idle-set -D 1
cpupower frequency-set -g performance
echo 0 >/sys/devices/system/cpu/cpufreq/boost

I used the raw performance counters from https://github.com/deater/deterministic/ initially, but r5000cf didn't work too well I always got ticks mismatch error on SYSCALL: write. I changed it to r5300cf and now I only got this error that overshoots most of the time by 8 (sometimes 7):

[ERROR /home/edwin/tools/rr/src/ReplaySession.cc:505:guard_overshoot() errno: 0 'Success'] 
 -> Replay diverged.  Dumping register comparison.
[FATAL /home/edwin/tools/rr/src/ReplaySession.cc:521:guard_overshoot() errno: 0 'Success'] 
 (task 11743 (rec:11732) at time 5675)
 ->  Assertion `false' failed to hold. overshot target ticks=548601495 by 8

Patch to rr looks like this currently (xsave register area is also too large, because value from ecx is used instead of ebx): https://gist.github.com/edwintorok/34a83868f7fb75d50937

I've run the tool from the deterministic project above with those counters, and the results are: https://gist.github.com/edwintorok/48b767542b50cdbb3b34. The only counter that looks deterministic is the retired MMX/FP instructions but I don't see how that could be used for rr.

Do the counters have to be absolutely deterministic for rr to work or are those counters with +- 12 errors on taken branches still usable?

[pipcet]

I believe there might yet be hope for AMD users: newish AMD CPUs implement something called Light-Weight Profiling (indicated by lwp in the CPUID output), as described in http://support.amd.com/TechDocs/43724.pdf. (Do not be distraught upon reading in the spec that there is a mechanism for randomizing the counters slightly when they reset; that can be disabled, in fact, by writing 0 to the relevant bits). I've made the minor modification necessary to the kernel to enable it, and so far the numbers look deterministic!

I haven't made the further modifications necessary to the kernel to enable LWP interrupts; do we need interrupts, or just a reliable monitoring counter? I haven't looked at the rr source code at all, yet, so any help would be appreciated.

[khuey]

rr requires interrupts for replay, yes. The code is pretty localized in src/PerfCounters.cc, you should be able to find the relevant bits there.

[rocallahan]

Somehow I missed replying to this:

Do the counters have to be absolutely deterministic for rr to work or are those counters with +- 12 errors on taken branches still usable?

Tolerating even small and infrequent errors is very difficult. I'm not saying it can't be done, but it makes rr more complicated and less reliable.

[rocallahan]

The caveats under the "gather" and "store" phases in the LWP document are OK as long as counting remains deterministic even when event records are skipped; it's not 100% clear to me that that's the case.

You might want to set the NAB and NRB control bits so that (AFAICT) only conditional branches are counted. This should make the AMD results consistent with what we get on Intel, which would help debugging (though not strictly necessary for correct operation). It would probably make the counters more robust by reducing the number of instructions eligible for counting.

It does look like, if these counters are really deterministic and subject to the caveats above, this could work!

[Keno]

A good first step to determining whether counts are deterministic is to try https://github.com/deater/deterministic, although even if that gives a clean bill of health, there may still be lingering issues (e.g. Silvermont passes that test suite but doesn't work with rr).

[pipcet]

Thanks for all the advice! I'm still trying to figure out whether this is worth doing properly. So far I've enabled interrupts in the kernel, to be delivered to the tracee (is that right?), hacked PerfCounters.cc, added an llwpcb to preload/preload.c, and appear to be able to forward-step through code with deterministic counter values at the same RIP.

One problem that might not be solvable is that LWP isn't transparent to the profiled process; it lives in process-accessible memory (not a problem if the tracee's well-behaved), and it plays havoc with XSAVE/XRSTOR: I think it actually gets disabled for signal handlers, then reenabled when those XRESTOR after exitting. So tracee code that uses XSAVE/XRSTOR will never work fully, and tracee code that has significant portions in signal handlers might require some kernel hacking.

The state now is that a very simple ni/rni pair of commands appears to be working, for code that doesn't use signal handlers (or indeed perform syscalls). Here's an rr log:

(rr) ni
0x00000000004004bc  5     while (++x);
1: x = 3
(rr) rni
0x00000000004004b8  5     while (++x);
1: x = 2
(rr) ni
0x00000000004004bc  5     while (++x);
1: x = 3
(rr) rni
0x00000000004004b8  5     while (++x);
1: x = 2
(rr) rc
Continuing.

Breakpoint 1, main () at /home/pip/longrun.c:3
3     volatile short x = 0;
1: x = 0
(rr) ni 1000
0x00000000004004bc  5     while (++x);
1: x = 200
(rr) rni 500
0x00000000004004bc  5     while (++x);
1: x = 100
(rr) 

That looks right to me.

[pipcet]

So I tried a few more things, and so far they appear to be working. For reference (and certainly not for review), these are the patches I'm using:

https://github.com/mozilla/rr/compare/master...pipcet:rr-amd?expand=1 https://github.com/torvalds/linux/compare/master...pipcet:rr-amd?expand=1

The kernel changes are minimal (I don't have a dedicated machine for testing right now, so I had to keep down the number of reboots): add storage for the LWP XFEATURE, and forward interrupts to a hard-coded interrupt vector (APIC vector 0xff for now) to the current process as SIGSTKFLT with code 0xdeadbeef.

The rr changes are a bit more significant: we're disabling the update of the extra registers (which would overwrite the LWP state), as well as performing some transformations on the ticks value we read that I confess I don't fully understand: the AMD performance counters count down, not up, and they do so in signed 26-bit integers, so we need to take care of wrap-around.

[rocallahan]

Sounds good!

I think it actually gets disabled for signal handlers

If true, that'll need to be fixed.

ExtraRegisters should be able to work with the modified XSAVE layout. The XSAVE area has a header with bits indicating which XSAVE features are present, and you can issue CPUID calls to determine the offset and size of each feature. You will need to do something so that saving the ExtraRegisters preserves LWP state.

So tracee code that uses XSAVE/XRSTOR will never work fully

I think you should be able to configure XCR0 in the kernel to disable save/restore of the LWP area from user space, which would fix that.

[rocallahan]

I don't know of any code that uses XSAVE/XRSTOR from user space, so it may not matter anyway.

[Keno]

The AMD documentation says Enable LWP by executing XSETBV to set bit 62 of XCR0. which may mean that LWP is not working without that XCR0 bit set.

I don't know of any code that uses XSAVE/XRSTOR from user space, so it may not matter anyway.

I have some.

[rocallahan]

Hmm. AFAICT if that's true then AMD designed LWP in a way that only works with user-space XSAVE/XRSTOR if the latter is aware of LWP, which would be ... unfortunate.

[Keno]

It's not a particularly unusual design, e.g. AVX instructions don't work unless the corresponding bit is set in XCR0.

[pipcet]

I think it actually gets disabled for signal handlers

If true, that'll need to be fixed.

I've yet to test this. For now, things seem to Just Work, which is pretty unexpected!

You will need to do something so that saving the ExtraRegisters preserves LWP state.

Sure, I think I'll just turn set_extra_regs into a read-modify-write thing. That should work, right?

I don't think the AMD documentation is clear on what happens if you re-enable LWP with the XCR0 bit set, then clear the bit before returning to userspace. Keno, is there code you can share which uses XRSTOR?

[rocallahan]

Sure, I think I'll just turn set_extra_regs into a read-modify-write thing. That should work, right?

Yes.

[rocallahan]

It would be good to add a test for user-space XSAVE/XRSTOR to rr's test suite.

[Keno]

Sure, it's not particularly isolated though: https://github.com/Keno/Gallium.jl/blob/master/src/Hooking/jumpto-x86_64-elf.s#L11

I'd happily put together a minimal assembly example. It's also worth noting that this particular one is not a problem, since it only selects exactly the state components it wants. The more problematic issue is that people sometimes just select all available ones and then get surprised when things break.

[pipcet]

Oh, and it turns out that writing the LWP area of the XSAVE image isn't enough, we also have to manually set the LWP bit in the XSAVE header area at offset 512 (well, 519). I'm not sure whether Intel documents the XSAVE header area, but AMD doesn't appear to.

With those changes, the first 132 (of 1708) tests pass. Test grandchild_threads_main_running.c does not, but I think that's because it has an endless loop involving an unconditional branch (https://github.com/mozilla/rr/blob/master/src/test/grandchild_threads_main_running.c#L32), which we don't count. (Does that test work on Intel?).

One thing I've been thinking about is whether we really have to set up LWP in the preload library; it would be nicer to use ptrace to force the first syscall to be an anonymous mmap() (to allocate the LWP control objects) and then execute llwpcb or xrstor, so we can start recording earlier.

[Keno]

I'm not sure whether Intel documents the XSAVE header area, but AMD doesn't appear to.

Yes, the XSAVE are is documented in Volume 1 the Intel architecture manual [1] Chapter 13

Test grandchild_threads_main_running.c does not

In that test the main thread gets killed, so the infinite loop is irrelevant.

I'm surprised all of this is even necessary. Is LWP not integrated with perf_events?

[1] http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-1-manual.pdf

[pipcet]

Yes, the XSAVE are is documented in Volume 1 the Intel architecture manual [1] Chapter 13

Thanks!

In that test the main thread gets killed, so the infinite loop is irrelevant.

Okay, I'll investigate that failure further, then.

I'm surprised all of this is even necessary. Is LWP not integrated with perf_events?

As far as I can tell, it isn't. It's wholly independent of the MSR-based performance monitoring, and what little kernel support there once was was removed in 2015 (http://www.gossamer-threads.com/lists/linux/kernel/2245025?page=last).

Do feel free to point me the right way if I'm wrong.

[Keno]

It seems like there was a good amount of work by AMD to integrate LWP: http://lkml.iu.edu/hypermail/linux/kernel/1112.2/00224.html (and follow up emails). Looks live it never went anywhere, but maybe a good place to start?

[pipcet]

Thanks for the link. I do not see the advantage of integrating LWP with the perf_events infrastructure as closely as that patch series appears to do, but, if I can get this working reliably enough to be useful, I'll ask about it on LKML.

(I think the best design would be for the number of LWP interrupts, but not the actual counter values, to be a perf_event. That would keep the kernel code down to a minimum (enable LWP, forward interrupts, no meddling in userspace memory), but allow us to share most of the code in PerfCounters.cc with the Intel version.)

[Keno]

Not suggesting any particular course of action, just thought maybe that would be a good place to start.

[rocallahan]

Does that test work on Intel?

Yes, all tests pass on Intel for me. With the 4.7rc5 kernel, I'm not even observing any intermittent failures.

One thing I've been thinking about is whether we really have to set up LWP in the preload library; it would be nicer to use ptrace to force the first syscall to be an anonymous mmap() (to allocate the LWP control objects) and then execute llwpcb or xrstor, so we can start recording earlier.

Sorry, I should have spotted this earlier. Yes, you really should start counting immediately after execve and clone. You can add code and read-only data to the rr_page to help with this (see generate_rr_page.py), and use ptrace to drive execution of the tracee task. AutoRemoteSyscall will help.

[pipcet]

Thanks again! I think I'm making progress:

• kernel code now uses perf_event
• counting starts right after exec()
• started writing shared code for PerfCounters.cc

Remaining issues:

• I'm sometimes seeing very large overshoots. I guess we could increase SKID_SIZE to something like 1500 to catch those, but I'd rather find out what is happening.
• some test suite failures haven't gone away
• no 32-bit support yet
• no test cases for XSAVE/XRSTOR (which should fail on AMD, of course)

Is it okay for SKID_SIZE to become variable? I feel really uncomfortable about increasing it by so much, and there's no reason Intel CPUs should take the performance and stability hit of having SKID_SIZE set way too high.

Are you testing with /proc/sys/kernel/randomize_va_space set to 1? I'm seeing failure in checksum_sanity only if that flag is set.

Does that test work on Intel?

Yes, all tests pass on Intel for me. With the 4.7rc5 kernel, I'm not even observing any intermittent failures.

Okay, that's great. I'll try working out what causes my remaining 64-bit test suite failures. All the 32-bit tests fail, because the 32-bit syscall sequence uses SYSCALL32, not SYSENTER, so it can't be patched (yet).

Sorry, I should have spotted this earlier. Yes, you really should start counting immediately after execve and clone. You can add code and read-only data to the rr_page to help with this (see generate_rr_page.py), and use ptrace to drive execution of the tracee task. AutoRemoteSyscall will help.

I'm starting the counts right after map_rr_page now, by mapping some anonymous memory (three more pages, to keep things page-aligned) right after the RR page, then modifying the XSAVE area. No more preload code. I think that should work as well as we can get this to work, as long as the tracee is well-behaved. (I still think it's a pity we have to make assumptions about that, but the LWP architecture really does not allow observing uncooperative tracees (I'm not sure what happens on Intel if a tracee attempts to munmap the RR page, but at least in theory we can catch that)).

But basically, this appears to work, and while the limitations of LWP are unfortunate, I think they're not relevant to most users.

[rocallahan]

Are you testing with /proc/sys/kernel/randomize_va_space set to 1? I'm seeing failure in checksum_sanity only if that flag is set.

2 actually.

Is it okay for SKID_SIZE to become variable? I feel really uncomfortable about increasing it by so much, and there's no reason Intel CPUs should take the performance and stability hit of having SKID_SIZE set way too high.

1500 sounds really high. It might be worth writing a standalone testcase to test the interrupt latency more thoroughly.

I think that should work as well as we can get this to work, as long as the tracee is well-behaved.

We don't try to handle malicious tracees.

[pipcet]

Okay, I'm down to 10 non-intermittent failures with ASLR disabled and the first version of the kernel patch: record_replay, term_trace_cpu, and mmap_shared_prot.

I've put together a test case for the interrupt latency, but I've yet to make sense of the results: usually the latency is 1, but I see occasional very high latencies distributed to about 2,000 retired conditional branches (correspondingly higher numbers of retired instructions), but no missed or double interrupts. The only thing I've found to actually keep down latencies is to modify the tracee loop's code to access the LWPCB and LWP buffer, which presumably keeps them in the cache and the translation information in the TLB somehow.

Is my understanding correct that this is only an issue during computation without syscalls? If so, would it be possible to recover from overshooting (assuming it's a rare event; I'm seeing about one interrupt in a million that exceeds 1,000 retired branches) by trying again? That might be a better strategy on my CPU than modifying SKID_SIZE.

[edwintorok]

On 2016-07-07 17:54, pipcet wrote:

Okay, I'm down to 10 non-intermittent failures with ASLR disabled and the first version of the kernel patch: record_replay, term_trace_cpu, and mmap_shared_prot.

Excellent progress, thanks for working on this!

I've put together a test case for the interrupt latency, but I've yet to make sense of the results: usually the latency is 1, but I see occasional very high latencies distributed to about 2,000 retired conditional branches

Did you try turning off cpuidle and cpupower (https://github.com/mozilla/rr/issues/1552#issuecomment-159998831)? It seems to make things a lot more predictable.

(correspondingly higher numbers of retired instructions), but no missed or double interrupts. The only thing I've found to actually keep down latencies is to modify the tracee loop's code to access the LWPCB and LWP buffer, which presumably keeps them in the cache and the translation information in the TLB somehow.

Is my understanding correct that this is only an issue during computation without syscalls? If so, would it be possible to recover from overshooting (assuming it's a rare event; I'm seeing about one interrupt in a million that exceeds 1,000 retired branches) by trying again? That might be a better strategy on my CPU than modifying SKID_SIZE.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mozilla/rr/issues/1552#issuecomment-231102577, or mute the thread https://github.com/notifications/unsubscribe/AAsD5jgPyJUI0pDO2FL5TsNGEjol5CFDks5qTRMsgaJpZM4GVB3g.

[pipcet]

Excellent progress, thanks for working on this!

Thank you! Are you (or is anyone else) in a position to test things while this still requires patching the kernel? So far I've only tested things on this one CPU, so I'd be happy to provide patches for testing on other AMD CPUs.

Did you try turning off cpuidle and cpupower (#1552 (comment))? It seems to make things a lot more predictable.

Thanks for the suggestion!

Disabling both power-saving states with cpuidle -D 0 does change the interrupt latency distribution significantly, which surprised me: with the power states disabled, about half of the time I'm seeing no interrupt latency at all; with them enabled, it's much more common for the latency to be at least one instruction.

It's too early to say whether this prevents the high latencies on my CPU entirely, but things are looking good, so far!

[edwintorok]

Would this work inside a virtual machine (on KVM with all the host CPU features enabled), or should I test bare-metal? I can do some tests, but only during the weekend, if the patch applies on top of 4.6.3-gentoo, so expect an ~1week lag between you posting the patch and me testing it.

[pipcet]

Would this work inside a virtual machine (on KVM with all the host CPU features enabled), or should I test bare-metal?

Bare-metal, I'm afraid. I haven't even looked at virtualizing LWP yet, but as it's implemented in a completely different way from ordinary PMCs, I'd be surprised if it were to work without extra code.

I'll try to get a patch to you tomorrow. Thanks again for the suggestion to use cpupower idle-set -D 0 (I got that command wrong in the last post), that really seems to prevent the extremely high latencies entirely.

[rocallahan]

Okay, I'm down to 10 non-intermittent failures with ASLR disabled and the first version of the kernel patch: record_replay, term_trace_cpu, and mmap_shared_prot.

Very promising!

f so, would it be possible to recover from overshooting (assuming it's a rare event; I'm seeing about one interrupt in a million that exceeds 1,000 retired branches) by trying again?

In theory yes, at significant performance and complexity cost. Increasing the skid size to 2000 would be preferable, but hopefully you've found a better solution.

Thanks again for the suggestion to use cpupower idle-set -D 0

Assuming we take the path of requiring this setting, we should have code in rr to detect it and bail out if it's not set correctly.

Thanks!

[pipcet]

record_replay, term_trace_cpu, and mmap_shared_prot.

Concerning the mmap_shared_prot test, I've found it necessary to apply https://github.com/pipcet/rr/commit/a67d40f4cb98a49375da951cadb2eb3bdac7a4f1. That change makes sense to me, since the number of trace records we read shouldn't depend on whether we call apply_mprotect_records once or twice (if stopped between the two mprotect() calls). But I don't know, if Intel is fine without the change that might be a red herring.

I "fixed" term_trace_cpu by replacing an xptrace by a ptrace_if_alive, but that's probably not the right thing to do, in general, since we don't want to pretend we've read data from the tracee when it's actually dead.

That leaves record_replay, which I think is a WONTFIX, unless there is a way to virtualize XSAVE/XRSTOR: it's possible to use the software-reserved LWP area to communicate between two processes both wanting to use LWP, but it's unnecessary complexity.

Next up is getting things to work without relying on the under-documented "continuous wrapping mode" feature, then hopefully fixing the perf_event kernel driver.

[rocallahan]

Concerning the mmap_shared_prot test, I've found it necessary to apply pipcet@a67d40f.

Your change looks correct. Thanks! I only recently added that so it's probably just a latent regression.

That leaves record_replay, which I think is a WONTFIX, unless there is a way to virtualize XSAVE/XRSTOR: it's possible to use the software-reserved LWP area to communicate between two processes both wanting to use LWP, but it's unnecessary complexity.

I'm OK with making that test WONTFIX for AMD. However, on Intel we do not use two counters; instead, the outer rr virtualizes perf_event_open etc for the inner rr so the inner rr is actually piggybacking on the outer rr's counter. So, it might not actually be too much work to special-case "inner rr on AMD", treating it as some made-up fake architecture, so that instead of using LWP it uses a regular perf_event_open that behaves like the Intel one and gets virtualized by the outer rr.

[pipcet]

Assuming we take the path of requiring this setting, we should have code in rr to detect it and bail out if it's not set correctly.

Agreed, though obviously that check should happen on AMD CPUs only. (I think we're actually fine with high interrupt latencies during recording, but checking for that setting only during replay has the potential to be very confusing to users).

So, it might not actually be too much work to special-case "inner rr on AMD", treating it as some made-up fake architecture, so that instead of using LWP it uses a regular perf_event_open that behaves like the Intel one and gets virtualized by the outer rr.

Nice idea. That should work, yes. I'm not going to do that right now, but it's a good option to have if we need rr-inside-rr to work for some reason.

[rocallahan]

Concerning the mmap_shared_prot test, I've found it necessary to apply pipcet@a67d40f.

I've landed my own version of this: https://github.com/mozilla/rr/commit/6140453e492cf3e136486bf245fddc9ef493c001 Let me know if it works for you.

[pipcet]

Let me know if it works for you.

Works fine, thanks!

[pipcet]

@edwintorok: https://github.com/torvalds/linux/compare/v4.6...pipcet:rr-amd-4.6.diff?diff=unified&name=rr-amd-4.6 is a raw diff against Linux 4.6, which hopefully applies to your tree, too. I'd recommend trying it with the rr-amd tree at https://github.com/pipcet/rr/tree/rr-amd, but I can also provide standalone test cases if that doesn't work right away.

[edwintorok]

@pipcet thanks, I built a new kernel and loaded your LWP driver:

# sysctl -w kernel.perf_event_paranoid = 1
# modprobe lwp
# dmesg|tail -n1
[  882.889133] Light-Weight Profiling Events: LWP driver.

I've built rr from your rr-amd branch (7120f136a7c12d61b8e18d99c825ff9aca11c643). How can I check that it is really using the LWP driver?

I added my CPU here (it is family 16h, but for this test I'll have it pretend its like yours):

     case 0x30f00:
+    case 0xf20:
       return AMDFamily15h;

I must be doing something wrong cause I get some failures on the alarm test:

bash  basic_test.run -b ~/rr/rr '' alarm
./util.sh: line 193: 32767 Aborted                 _RR_TRACE_DIR="$workdir" rr $GLOBAL_OPTIONS record $LIB_ARG $RECORD_ARGS $exe $exeargs > record.out 2> record.err
./util.sh: line 219:   314 Aborted                 _RR_TRACE_DIR="$workdir" rr $GLOBAL_OPTIONS replay -a $replayflags > replay.out 2> replay.err
Test 'alarm' FAILED: : error during recording:
--------------------------------------------------
[ERROR /home/edwin/rr/rr/src/AddressSpace.cc:1179:assert_segments_match() errno: SUCCESS] cached mmap:
--------------------------------------------------
record.out:
--------------------------------------------------
--------------------------------------------------
Test alarm failed, leaving behind /tmp/rr-test-alarm-42ufhtVX3
To replay the failed test, run
  _RR_TRACE_DIR=/tmp/rr-test-alarm-42ufhtVX3 rr replay

If I remove the mmap flag from util.sh I get this:

bash  basic_test.run -b ~/rr/rr '' alarm
./util.sh: line 219:   372 Aborted                 _RR_TRACE_DIR="$workdir" rr $GLOBAL_OPTIONS replay -a $replayflags > replay.out 2> replay.err
Test 'alarm' FAILED: : error during replay:
--------------------------------------------------
[FATAL /home/edwin/rr/rr/src/ReplaySession.cc:833:check_ticks_consistency() errno: SUCCESS] 
 (task 373 (rec:371) at time 15)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SEGV_RDTSC'; expected 250355, got 16777215
[FATAL /home/edwin/rr/rr/src/log.cc:270:emergency_debug() errno: SUCCESS] (session doesn't look interactive, aborting emergency debugging)
--------------------------------------------------

If I run rr record -b alarm and rr replay manually I get this:

[FATAL /home/edwin/rr/rr/src/ReplaySession.cc:833:check_ticks_consistency() errno: SUCCESS] 
 (task 22659 (rec:22610) at time 15)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SEGV_RDTSC'; expected 416688, got 16777215
Launch gdb with 
  gdb '-l' '-1' '-ex' 'target extended-remote :22659' /home/edwin/.local/share/rr/latest-trace/mmap_hardlink_2_alarm

$ gdb '-l' '-1' '-ex' 'target extended-remote :22659' /home/edwin/.local/share/rr/latest-trace/mmap_hardlink_2_alarm
GNU gdb (Gentoo 7.11.1 vanilla) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/edwin/.local/share/rr/latest-trace/mmap_hardlink_2_alarm...done.
Remote debugging using :22659
warning: remote target does not support file transfer, attempting to access files from local filesystem.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib64/debug//lib64/ld-2.23.so.debug...done.
done.
_dl_start (arg=0x7fff5969fe00) at rtld.c:355
355     HP_TIMING_NOW (start_time);
(gdb) bt
#0  _dl_start (arg=0x7fff5969fe00) at rtld.c:355
#1  0x00007fa206b63c88 in _start () from /lib64/ld-linux-x86-64.so.2
#2  0x0000000000000001 in ?? ()
#3  0x00007fff596a1a78 in ?? ()
#4  0x0000000000000000 in ?? ()

And here's the error for the mmap (is the problem the leading 0x?)

~/rr/obj/bin/rr record --check-cached-mmaps -b ../../../obj/bin/alarm
rr: Saving execution to trace directory `/home/edwin/.local/share/rr/alarm-1'.
[ERROR /home/edwin/rr/rr/src/AddressSpace.cc:1179:assert_segments_match() errno: SUCCESS] cached mmap:
  (heap: 0x19d4000-0x19d4000)
0x400000-0x402000 r-xp 00000000 fd:02 3183248    /home/edwin/rr/obj/bin/alarm
0x601000-0x603000 rw-p 00001000 fd:02 3183248    /home/edwin/rr/obj/bin/alarm
0x68000000-0x68200000 rwxp 00000000 00:00 0          
0x70000000-0x70001000 r-xp 00000000 fd:02 3182361    /home/edwin/rr/obj/bin/rr_page_64
0x70001000-0x70004000 rw-p 00000000 fd:02 3182361    
0x7fdc5c27b000-0x7fdc5c29e000 r-xp 00000000 fd:01 1445885    /lib64/ld-2.23.so
0x7fdc5c49d000-0x7fdc5c49f000 rw-p 00022000 fd:01 1445885    /lib64/ld-2.23.so
0x7fdc5c49f000-0x7fdc5c4a0000 rw-p 00000000 00:00 0          
0x7ffc96652000-0x7ffc96675000 rw-p 00000000 00:00 0          [stack]
0x7ffc9672e000-0x7ffc96730000 r-xp 00000000 00:00 0          [vdso]
[ERROR /home/edwin/rr/rr/src/AddressSpace.cc:1181:assert_segments_match() errno: SUCCESS] /proc/22717/mmaps:
00400000-00402000 r-xp 00000000 fd:02 3183248                            /home/edwin/rr/obj/bin/alarm
00601000-00603000 rw-p 00001000 fd:02 3183248                            /home/edwin/rr/obj/bin/alarm
68000000-68200000 rwxp 00000000 00:00 0 
70000000-70001000 r-xp 00000000 fd:02 3182361                            /home/edwin/rr/obj/bin/rr_page_64
70001000-70004000 rw-p 00000000 00:00 0 
7fdc5c27b000-7fdc5c29e000 r-xp 00000000 fd:01 1445885                    /lib64/ld-2.23.so
7fdc5c49d000-7fdc5c49f000 rw-p 00022000 fd:01 1445885                    /lib64/ld-2.23.so
7fdc5c49f000-7fdc5c4a0000 rw-p 00000000 00:00 0 
7ffc96652000-7ffc96675000 rw-p 00000000 00:00 0                          [stack]
7ffc9672e000-7ffc96730000 r-xp 00000000 00:00 0                          [vdso]
[FATAL /home/edwin/rr/rr/src/AddressSpace.cc:1183:assert_segments_match() errno: SUCCESS] 
 (task 22717 (rec:22717) at time 14)
 -> Assertion `false' failed to hold. 

Note: I'm on gentoo now, and have the system compiled with:

CFLAGS="-march=native -O2 -mno-tbm -pipe -frecord-gcc-switches"
CXXFLAGS="${CFLAGS}"
CPU_FLAGS_X86="aes avx fma3 fma4 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop"

[pipcet]

Thank you very much for testing so quickly.

To answer the second question first, I don't think the issue is the leading 0x. Rather, the segment for the 70001000-70004000 region (that's the LWP control block and data area) doesn't match. That's currently to be expected, I haven't gotten around to excluding those areas from being checked for equality yet. So that's all as it should be.

The first error, however, is not. The first ticks value is realistic, so LWP appears to have worked during recording, but the second value looks like the maximum ticks value has been replaced by 0 in memory.

I must be doing something wrong cause I get some failures on the alarm test:

No, it's quite probably me who did something wrong. Can you check the output of rdmsr -a 0xc0000105 (after inserting the msr module, if you don't have that built into your kernel)?

[edwintorok]

On 2016-07-10 12:54, pipcet wrote:

|rdmsr -a 0xc0000105|

Here you go:

rdmsr -a 0xc0000105 0 0 0 0 0 0 0 0

[pipcet]

I'm a bit confused about your CPU, now. The one you posted the cpuinfo for above is a family 15h, as far as I can tell, and LWP support is present on that CPU.

If you still have time, can you try running perf stat -e /lwp/lwp-interrupts/u rr record ./alarm or similar? That should abort with an error message, and the "stats" should show a small number (1 or 2 here) of interrupts, but not zero. I did find some minor bugs that I don't think explain the failures you see, will push to the rr-amd branch in a few minutes.

[edwintorok]

On 07/10/2016 01:45 PM, pipcet wrote:

I'm a bit confused about your CPU, now. The one you posted the cpuinfo for above is a family 15h, as far as I can tell, and LWP support is present on that CPU.

Sorry, was confused that it didn't match the AMDFamily15h switch case. cpu family: 21 = 15h.

If you still have time, can you try running |perf stat -e /lwp/lwp-interrupts/u rr record ./alarm| or similar? That should abort with an error message, and the "stats" should show a small number (1 or 2 here) of interrupts, but not zero. I did find some minor bugs that I don't think explain the failures you see, will push to the |rr-amd| branch in a few minutes.

This causes the system to become unresponsive and print messages like this: watchdog detected hard LOCKUP on cpu 4. I'll try to see if I can capture those messages and post them here.

[edwintorok]

On 2016-07-10 14:11, Török Edwin wrote:

On 07/10/2016 01:45 PM, pipcet wrote:

I'm a bit confused about your CPU, now. The one you posted the cpuinfo for above is a family 15h, as far as I can tell, and LWP support is present on that CPU.

Sorry, was confused that it didn't match the AMDFamily15h switch case. cpu family: 21 = 15h.

If you still have time, can you try running |perf stat -e /lwp/lwp-interrupts/u rr record ./alarm| or similar? That should abort with an error message, and the "stats" should show a small number (1 or 2 here) of interrupts, but not zero. I did find some minor bugs that I don't think explain the failures you see, will push to the |rr-amd| branch in a few minutes.

This causes the system to become unresponsive and print messages like this: watchdog detected hard LOCKUP on cpu 4. I'll try to see if I can capture those messages and post them here.

Trouble starts here: https://gist.github.com/anonymous/5516faedee6ea81fa7a175d49a270ebc#file-dmesg-L988

[ 141.022375] general protection fault: 0000 [#1] SMP [ 141.023058] Modules linked in: lwp msr cpuid tun bridge stp llc ebtable_filter ebtables bnep bluetooth iptable_raw xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_REJECT nf_reject_ipv6 xt_addrtype nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_conntrack nf_nat nf_conntrack ip6table_filter ip6_tables ip_tables snd_hda_codec_hdmi it87 hwmon_vid snd_hda_codec_realtek snd_hda_codec_generic kvm_amd radeon kvm irqbypass crc32c_intel i2c_algo_bit ghash_clmulni_intel ttm cryptd snd_hda_intel edac_mce_amd drm_kms_helper serio_raw pcspkr edac_core syscopyarea k10temp fam15h_power snd_hda_codec sysfillrect sysimgblt fb_sys_fops snd_hda_core snd_hwdep drm snd_pcm snd_timer snd i2c_core soundcore ohci_pci [ 141.025574] wmi acpi_cpufreq dm_zero dm_thin_pool dm_persistent_data dm_bio_prison dm_service_time dm_round_robin dm_queue_length dm_multipath dm_log_userspace dm_flakey dm_delay xts gf128mul cbc sha256_generic scsi_transport_iscsi r8169 mii fuse xfs nfs lockd grace sunrpc fscache ext2 raid10 raid1 raid0 dm_raid raid456 async_memcpy async_raid6_recov libcrc32c async_pq raid6_pq async_xor xor async_tx dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd sr_mod cdrom sg [ 141.028363] CPU: 0 PID: 4612 Comm: perf Not tainted 4.6.3-gentoo-rr #3 [ 141.029322] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2501 04/07/2014 [ 141.030299] task: ffff8800b18d4a80 ti: ffff880317f48000 task.ti: ffff880317f48000 [ 141.031288] RIP: 0010:[] [] refresh_lwp_cfg+0x1d/0x60 [lwp] [ 141.032297] RSP: 0018:ffff880317f4bc58 EFLAGS: 00010017 [ 141.033277] RAX: 00000000e000000e RBX: ffff88033ec16720 RCX: 00000000c0000105 [ 141.034290] RDX: 000000000000ff00 RSI: 0000ff00e000000e RDI: ffffffffc071f100 [ 141.035302] RBP: ffff880317f4bc70 R08: 0000000000000001 R09: 00000000ffffffff [ 141.036317] R10: 0000000000000000 R11: 00000000000003db R12: ffff88033ec16720 [ 141.037338] R13: ffff8800b18d4a80 R14: ffffffff81c0b500 R15: ffff8800b18d56e0 [ 141.038360] FS: 00007f69fae0c780(0000) GS:ffff88033ec00000(0000) knlGS:0000000000000000 [ 141.039355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 141.040394] CR2: 00000000007750e0 CR3: 00000000b2fee000 CR4: 00000000000406f0 [ 141.041422] Stack: [ 141.042423] ffffffffc071d470 ffff8800b18d56d8 ffff8800ba3f4e00 ffff880317f4bc80 [ 141.043444] ffffffff811436d7 ffff880317f4bce0 ffffffff81145b4d ffff88033ec18b20 [ 141.044462] ffffffff8108a5df 0000000000000000 00000000fffd95db ffff880317f4bd18 [ 141.045459] Call Trace: [ 141.046481] [] ? lwp_enable+0x60/0x80 [lwp] [ 141.047523] [] perf_pmu_enable.part.50+0x7/0x10 [ 141.048567] [] perf_event_task_sched_in+0x19d/0x1b0 [ 141.049621] [] ? sched_clock_cpu+0x8f/0xa0 [ 141.050682] [] finish_task_switch+0x171/0x210 [ 141.051743] [] schedule+0x39b/0xa30 [ 141.052772] [] schedule+0x35/0x80 [ 141.053831] [] pipe_wait+0x61/0x90 [ 141.054895] [] ? wake_atomic_t_function+0x70/0x70 [ 141.055965] [] pipe_read+0x216/0x300 [ 141.057032] [] ? file_has_perm+0x8c/0xb0 [ 141.058103] [] __vfs_read+0xa7/0xd0 [ 141.059169] [] vfs_read+0x95/0x140 [ 141.060232] [] SyS_read+0x4f/0xb0 [ 141.061264] [] entry_SYSCALL_64_fastpath+0x18/0xa3 [ 141.062328] Code: 44 00 00 55 b8 01 00 00 00 48 89 e5 5d c3 0f 1f 44 00 00 48 85 ff 74 08 48 8b 37 48 85 f6 75 25 31 c0 31 d2 31 f6 b9 05 01 00 c0 <0f> 30 0f 1f 44 00 00 c3 55 31 d2 bf 05 01 00 c0 48 89 e5 e8 2b [ 141.063533] RIP [] refresh_lwp_cfg+0x1d/0x60 [lwp] [ 141.064661] RSP [ 141.065785] ---[ end trace ba263e919464dd09 ]---

.. then bunch of NMI errors if I try to kill -9 perf.

And here is my .config, is there anything here not compatible with LWP? https://gist.github.com/anonymous/a71ca0e86698f0b814d18d85557d3c73

Thanks,

[pipcet]

Oh, I see. You don't have "continuous mode" sampling or the PTSC thing, I'd assumed your CPU would be newer than mine and have it :-)

Can you change 0xe000000e to 0x80000008 in arch/x86/events/amd/lwp.c and try recompiling? That should avoid the oops, at least.

[edwintorok]

On 2016-07-10 14:43, pipcet wrote:

Oh, I see. You don't have "continuous mode" sampling or the PTSC thing, I'd assumed your CPU would be newer than mine and have it :-)

Can you change 0xe000000e to 0x80000008 in arch/x86/events/amd/lwp.c and try recompiling? That should avoid the oops, at least.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/rr/issues/1552#issuecomment-231584759, or mute the thread https://github.com/notifications/unsubscribe/AAsD5rDXPEd5-rP8k7r30u5FpRDefEZLks5qUNr4gaJpZM4GVB3g.

Thanks, the oops is gone, and here's the output now:

$ perf stat -e /lwp/lwp-interrupts/u rr record ./alarm rr: Saving execution to trace directory /home/edwin/.local/share/rr/alarm-12'. ..rr: /home/edwin/rr/rr/src/PerfCounters.cc:353: Ticks rr::PerfCounters::read_ticks_nondestructively(): Assertioncounter_value >= -2048' failed. rr: Aborted / Performance counter stats for 'rr record ./alarm':

             2      lwp/lwp-interrupts/u                                        

   0.031624790 seconds time elapsed

$ perf stat -e /lwp/lwp-interrupts/u rr record ./alarm rr: Saving execution to trace directory /home/edwin/.local/share/rr/alarm-13'. .rr: /home/edwin/rr/rr/src/PerfCounters.cc:353: Ticks rr::PerfCounters::read_ticks_nondestructively(): Assertioncounter_value >= -2048' failed. rr: Aborted / Performance counter stats for 'rr record ./alarm':

             1      lwp/lwp-interrupts/u                                        

   0.031811759 seconds time elapsed

bash basic_test.run -b ~/rr/rr '' alarm ./util.sh: line 219: 4924 Aborted _RR_TRACE_DIR="$workdir" rr $GLOBAL_OPTIONS replay -a $replayflags > replay.out 2> replay.err

Test 'alarm' FAILED: : error during replay:

[FATAL /home/edwin/rr/rr/src/ReplaySession.cc:833:check_ticks_consistency() errno: SUCCESS](task 4925 %28rec:4923%29 at time 15) -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SEGV_RDTSC'; expected 27074, got 16777215

[FATAL /home/edwin/rr/rr/src/log.cc:270:emergency_debug() errno: SUCCESS](session doesn't look interactive, aborting emergency debugging)

replay.out:

---

Test alarm failed, leaving behind /tmp/rr-test-alarm-MxcgEhZ0x To replay the failed test, run _RR_TRACE_DIR=/tmp/rr-test-alarm-MxcgEhZ0x rr replay

And rdmsr: rdmsr -a 0xc0000105 0 0 0 0 0 0 ff0080000008 0

[edwintorok]

On 2016-07-10 14:51, Török Edwin wrote:

On 2016-07-10 14:43, pipcet wrote:

Oh, I see. You don't have "continuous mode" sampling or the PTSC thing, I'd assumed your CPU would be newer than mine and have it :-)

Can you change 0xe000000e to 0x80000008 in arch/x86/events/amd/lwp.c and try recompiling? That should avoid the oops, at least.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/rr/issues/1552#issuecomment-231584759, or mute the thread https://github.com/notifications/unsubscribe/AAsD5rDXPEd5-rP8k7r30u5FpRDefEZLks5qUNr4gaJpZM4GVB3g.

Thanks, the oops is gone, and here's the output now:

$ perf stat -e /lwp/lwp-interrupts/u rr record ./alarm rr: Saving execution to trace directory /home/edwin/.local/share/rr/alarm-12'. ..rr: /home/edwin/rr/rr/src/PerfCounters.cc:353: Ticks rr::PerfCounters::read_ticks_nondestructively(): Assertioncounter_value >= -2048' failed. rr: Aborted / Performance counter stats for 'rr record ./alarm':

             2      lwp/lwp-interrupts/u                                        

   0.031624790 seconds time elapsed

$ perf stat -e /lwp/lwp-interrupts/u rr record ./alarm rr: Saving execution to trace directory /home/edwin/.local/share/rr/alarm-13'. .rr: /home/edwin/rr/rr/src/PerfCounters.cc:353: Ticks rr::PerfCounters::read_ticks_nondestructively(): Assertioncounter_value >= -2048' failed. rr: Aborted / Performance counter stats for 'rr record ./alarm':

             1      lwp/lwp-interrupts/u                                        

   0.031811759 seconds time elapsed

bash basic_test.run -b ~/rr/rr '' alarm ./util.sh: line 219: 4924 Aborted _RR_TRACE_DIR="$workdir" rr $GLOBAL_OPTIONS replay -a $replayflags > replay.out 2> replay.err

Test 'alarm' FAILED: : error during replay:

[FATAL /home/edwin/rr/rr/src/ReplaySession.cc:833:check_ticks_consistency() errno: SUCCESS](task 4925 %28rec:4923%29 at time 15) -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SEGV_RDTSC'; expected 27074, got 16777215

[FATAL /home/edwin/rr/rr/src/log.cc:270:emergency_debug() errno: SUCCESS](session doesn't look interactive, aborting emergency debugging)

replay.out:

---

Test alarm failed, leaving behind /tmp/rr-test-alarm-MxcgEhZ0x To replay the failed test, run _RR_TRACE_DIR=/tmp/rr-test-alarm-MxcgEhZ0x rr replay

And rdmsr: rdmsr -a 0xc0000105 0 0 0 0 0 0 ff0080000008 0

Also I see this in dmesg now:

[ 129.980601] ------------[ cut here ]------------ [ 129.980609] WARNING: CPU: 1 PID: 4578 at arch/x86/events/amd/lwp.c:260 lwp_start+0x7c/0x80 [lwp] [ 129.980610] Modules linked in: cpuid msr lwp tun bridge stp llc ebtable_filter ebtables bnep bluetooth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 iptable_filter xt_conntrack xt_CHECKSUM nf_conntrack iptable_mangle ip6table_filter iptable_raw ip6_tables ip_tables snd_hda_codec_hdmi it87 hwmon_vid snd_hda_codec_realtek snd_hda_codec_generic kvm_amd radeon kvm snd_hda_intel irqbypass i2c_algo_bit snd_hda_codec ttm crc32c_intel snd_hda_core ghash_clmulni_intel drm_kms_helper snd_hwdep cryptd snd_pcm syscopyarea edac_mce_amd sysfillrect sysimgblt snd_timer serio_raw fb_sys_fops edac_core pcspkr snd k10temp fam15h_power drm soundcore i2c_core ohci_pci [ 129.980642] wmi acpi_cpufreq dm_zero dm_thin_pool dm_persistent_data dm_bio_prison dm_service_time dm_round_robin dm_queue_length dm_multipath dm_log_userspace dm_flakey dm_delay xts gf128mul cbc sha256_generic scsi_transport_iscsi r8169 mii fuse xfs nfs lockd grace sunrpc fscache ext2 raid10 raid1 raid0 dm_raid raid456 async_memcpy async_raid6_recov libcrc32c async_pq raid6_pq async_xor xor async_tx dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd sr_mod cdrom sg [ 129.980668] CPU: 1 PID: 4578 Comm: perf Not tainted 4.6.3-gentoo-rr #3 [ 129.980670] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2501 04/07/2014 [ 129.980671] 0000000000000000 ffff8800ae5dbb70 ffffffff81334970 0000000000000000 [ 129.980674] 0000000000000000 ffff8800ae5dbbb0 ffffffff8105c4bd 00000104ffffffff [ 129.980676] ffff88032e79d000 ffff88033ec56720 ffff88033ec58c10 ffff88033ec58c14 [ 129.980678] Call Trace: [ 129.980683] [] dump_stack+0x67/0x97 [ 129.980686] [] __warn+0xdd/0x100 [ 129.980689] [] warn_slowpath_null+0x1d/0x20 [ 129.980691] [] lwp_start+0x7c/0x80 [lwp] [ 129.980693] [] lwp_enable+0x45/0x80 [lwp] [ 129.980696] [] perf_pmu_enable.part.50+0x7/0x10 [ 129.980698] [] perf_pmu_commit_txn+0x3a/0x40 [ 129.980700] [] group_sched_in+0x1aa/0x1d0 [ 129.980702] [] ctx_sched_in+0x23f/0x4c0 [ 129.980704] [] perf_event_sched_in+0x6b/0xa0 [ 129.980706] [] ctx_resched+0x5b/0x90 [ 129.980708] [] perf_event_exec+0x123/0x150 [ 129.980710] [] setup_new_exec+0x6f/0x1d0 [ 129.980713] [] load_elf_binary+0x3bb/0x1150 [ 129.980715] [] ? load_misc_binary+0x64/0x3f0 [ 129.980717] [] ? get_user_pages_remote+0x54/0x60 [ 129.980719] [] search_binary_handler+0x9d/0x1d0 [ 129.980721] [] do_execveat_common.isra.16+0x51b/0x6e0 [ 129.980723] [] SyS_execve+0x3f/0x50 [ 129.980725] [] do_syscall_64+0x63/0xd0 [ 129.980728] [] entry_SYSCALL64_slow_path+0x25/0x25 [ 129.980730] ---[ end trace 5b483c7d64348560 ]--- [ 130.002534] lwp_interrupt, 15 active events [ 130.013044] lwp_interrupt, 15 active events [ 131.557999] lwp_interrupt, 15 active events [ 131.984290] lwp_interrupt, 15 active events [ 151.361052] lwp_interrupt, 15 active events [ 152.138919] lwp_interrupt, 15 active events [ 152.659839] lwp_interrupt, 15 active events [ 153.077454] lwp_interrupt, 15 active events [ 153.082578] lwp_interrupt, 15 active events [ 189.904012] lwp_interrupt, 15 active events [ 190.417537] lwp_interrupt, 15 active events [ 190.882078] lwp_interrupt, 15 active events

[pipcet]

Excellent, I left that in to see whether interrupts arrive. They do!

I'll have to have a look to see what's going on with the counter not starting, though. Strangely, when I record alarm and rr dump it, global_time = 15 corresponds to the last event that's happening at 0 ticks, so both numbers would be wrong. Do the ticks in the events that rr dump shows look plausible? Feel free to post a gist, of course.

[edwintorok]

On 2016-07-10 15:16, pipcet wrote:

Excellent, I left that in to see whether interrupts arrive. They do!

I'll have to have a look to see what's going on with the counter not starting, though. Strangely, when I record |alarm| and |rr dump| it, global_time = 15 corresponds to the last event that's happening at 0 ticks, so both numbers would be wrong. Do the ticks in the events that |rr dump| shows look plausible? Feel free to post a gist, of course.

The ticks grow until here, so that seems fine to me:

{
  real_time:3172.289675 global_time:2621, event:`EXIT' tid:5804, ticks:329150797
}

But the dump ends like this, which looks odd, I've expected something other than zero here:

{
  real_time:3172.289733 global_time:2622, event:`TRACE_TERMINATION' tid:0, ticks:0
}

Here is the 'rr dump' output (after running 'rr record ./alarm'): https://gist.githubusercontent.com/anonymous/b2000360376ba967add50b15df09bcf8/raw/3c6783463445f030d9f29161e47cb89787b752f5/gistfile1.txt