rr-debugger / rr

Record and Replay Framework
http://rr-project.org/
Other
9.12k stars 583 forks source link

Support AMD Piledriver? #1552

Closed edwintorok closed 7 years ago

edwintorok commented 8 years ago

I wanted to try rr but it looks like it doesn't support my AMD CPU:

[FATAL /home/edwin/rr/rr/src/PerfCounters.cc:129:get_cpu_microarch() errno: 0 'Success'] 
 -> CPU 0xf20 unknown.

The CPU is an AMD FX-8350, which is AFAICT Piledriver microarchitecture, see below for cpuinfo and perf list output. The available performance counter events should be documented in 3.15 Core Performance Counter Events. Is it possible to support rr on this microarchitecture, and what would be the value for struct PmuConfig?

$ dmesg|grep PMU
[    0.346218] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    0.346626] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
$ uname -a
Linux debian 4.2.0-0.bpo.1-amd64 #1 SMP Debian 4.2.3-2~bpo8+1 (2015-10-20) x86_64 GNU/Linux

/proc/cpuinfo for a core:

processor   : 7
vendor_id   : AuthenticAMD
cpu family  : 21
model       : 2
model name  : AMD FX(tm)-8350 Eight-Core Processor
stepping    : 0
microcode   : 0x6000832
cpu MHz     : 1400.000
cache size  : 2048 KB
physical id : 0
siblings    : 8
core id     : 7
cpu cores   : 4
apicid      : 23
initial apicid  : 7
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 8026.66
TLB size    : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

List of pre-defined events (to be used in -e):

branch-instructions OR branches [Hardware event] branch-misses [Hardware event] cache-misses [Hardware event] cache-references [Hardware event] cpu-cycles OR cycles [Hardware event] instructions [Hardware event] stalled-cycles-backend OR idle-cycles-backend [Hardware event] stalled-cycles-frontend OR idle-cycles-frontend [Hardware event]

alignment-faults [Software event] context-switches OR cs [Software event] cpu-clock [Software event] cpu-migrations OR migrations [Software event] dummy [Software event] emulation-faults [Software event] major-faults [Software event] minor-faults [Software event] page-faults OR faults [Software event] task-clock [Software event]

L1-dcache-load-misses [Hardware cache event] L1-dcache-loads [Hardware cache event] L1-dcache-prefetch-misses [Hardware cache event] L1-dcache-prefetches [Hardware cache event] L1-dcache-stores [Hardware cache event] L1-icache-load-misses [Hardware cache event] L1-icache-loads [Hardware cache event] L1-icache-prefetches [Hardware cache event] LLC-load-misses [Hardware cache event] LLC-loads [Hardware cache event] LLC-stores [Hardware cache event] branch-load-misses [Hardware cache event] branch-loads [Hardware cache event] dTLB-load-misses [Hardware cache event] dTLB-loads [Hardware cache event] iTLB-load-misses [Hardware cache event] iTLB-loads [Hardware cache event]

branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] branch-misses OR cpu/branch-misses/ [Kernel PMU event] cache-misses OR cpu/cache-misses/ [Kernel PMU event] cache-references OR cpu/cache-references/ [Kernel PMU event] cpu-cycles OR cpu/cpu-cycles/ [Kernel PMU event] instructions OR cpu/instructions/ [Kernel PMU event]

rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor](see 'man perf-list' on how to encode it)

mem:[/len][:access] [Hardware breakpoint]

block:block_bio_backmerge [Tracepoint event] block:block_bio_bounce [Tracepoint event] block:block_bio_complete [Tracepoint event] block:block_bio_frontmerge [Tracepoint event] block:block_bio_queue [Tracepoint event] block:block_bio_remap [Tracepoint event] block:block_dirty_buffer [Tracepoint event] block:block_getrq [Tracepoint event] block:block_plug [Tracepoint event] block:block_rq_abort [Tracepoint event] block:block_rq_complete [Tracepoint event] block:block_rq_insert [Tracepoint event] block:block_rq_issue [Tracepoint event] block:block_rq_remap [Tracepoint event] block:block_rq_requeue [Tracepoint event] block:block_sleeprq [Tracepoint event] block:block_split [Tracepoint event] block:block_touch_buffer [Tracepoint event] block:block_unplug [Tracepoint event] btrfs:__extent_writepage [Tracepoint event] btrfs:add_delayed_data_ref [Tracepoint event] btrfs:add_delayed_ref_head [Tracepoint event] btrfs:add_delayed_tree_ref [Tracepoint event] btrfs:alloc_extent_state [Tracepoint event] btrfs:btrfs_all_work_done [Tracepoint event] btrfs:btrfs_chunk_alloc [Tracepoint event] btrfs:btrfs_chunk_free [Tracepoint event] btrfs:btrfs_cow_block [Tracepoint event] btrfs:btrfs_failed_cluster_setup [Tracepoint event] btrfs:btrfs_find_cluster [Tracepoint event] btrfs:btrfs_get_extent [Tracepoint event] btrfs:btrfs_inode_evict [Tracepoint event] btrfs:btrfs_inode_new [Tracepoint event] btrfs:btrfs_inode_request [Tracepoint event] btrfs:btrfs_ordered_extent_add [Tracepoint event] btrfs:btrfs_ordered_extent_put [Tracepoint event] btrfs:btrfs_ordered_extent_remove [Tracepoint event] btrfs:btrfs_ordered_extent_start [Tracepoint event] btrfs:btrfs_ordered_sched [Tracepoint event] btrfs:btrfs_reserve_extent [Tracepoint event] btrfs:btrfs_reserve_extent_cluster [Tracepoint event] btrfs:btrfs_reserved_extent_alloc [Tracepoint event] btrfs:btrfs_reserved_extent_free [Tracepoint event] btrfs:btrfs_setup_cluster [Tracepoint event] btrfs:btrfs_space_reservation [Tracepoint event] btrfs:btrfs_sync_file [Tracepoint event] btrfs:btrfs_sync_fs [Tracepoint event] btrfs:btrfs_transaction_commit [Tracepoint event] btrfs:btrfs_work_queued [Tracepoint event] btrfs:btrfs_work_sched [Tracepoint event] btrfs:btrfs_workqueue_alloc [Tracepoint event] btrfs:btrfs_workqueue_destroy [Tracepoint event] btrfs:btrfs_writepage_end_io_hook [Tracepoint event] btrfs:find_free_extent [Tracepoint event] btrfs:free_extent_state [Tracepoint event] btrfs:run_delayed_data_ref [Tracepoint event] btrfs:run_delayed_ref_head [Tracepoint event] btrfs:run_delayed_tree_ref [Tracepoint event] cfg80211:cfg80211_cac_event [Tracepoint event] cfg80211:cfg80211_ch_switch_notify [Tracepoint event] cfg80211:cfg80211_ch_switch_started_notify [Tracepoint event] cfg80211:cfg80211_chandef_dfs_required [Tracepoint event] cfg80211:cfg80211_cqm_pktloss_notify [Tracepoint event] cfg80211:cfg80211_cqm_rssi_notify [Tracepoint event] cfg80211:cfg80211_del_sta [Tracepoint event] cfg80211:cfg80211_ft_event [Tracepoint event] cfg80211:cfg80211_get_bss [Tracepoint event] cfg80211:cfg80211_gtk_rekey_notify [Tracepoint event] cfg80211:cfg80211_ibss_joined [Tracepoint event] cfg80211:cfg80211_inform_bss_width_frame [Tracepoint event] cfg80211:cfg80211_mgmt_tx_status [Tracepoint event] cfg80211:cfg80211_michael_mic_failure [Tracepoint event] cfg80211:cfg80211_new_sta [Tracepoint event] cfg80211:cfg80211_notify_new_peer_candidate [Tracepoint event] cfg80211:cfg80211_pmksa_candidate_notify [Tracepoint event] cfg80211:cfg80211_probe_status [Tracepoint event] cfg80211:cfg80211_radar_event [Tracepoint event] cfg80211:cfg80211_ready_on_channel [Tracepoint event] cfg80211:cfg80211_ready_on_channel_expired [Tracepoint event] cfg80211:cfg80211_reg_can_beacon [Tracepoint event] cfg80211:cfg80211_report_obss_beacon [Tracepoint event] cfg80211:cfg80211_report_wowlan_wakeup [Tracepoint event] cfg80211:cfg80211_return_bool [Tracepoint event] cfg80211:cfg80211_return_bss [Tracepoint event] cfg80211:cfg80211_return_u32 [Tracepoint event] cfg80211:cfg80211_return_uint [Tracepoint event] cfg80211:cfg80211_rx_mgmt [Tracepoint event] cfg80211:cfg80211_rx_mlme_mgmt [Tracepoint event] cfg80211:cfg80211_rx_spurious_frame [Tracepoint event] cfg80211:cfg80211_rx_unexpected_4addr_frame [Tracepoint event] cfg80211:cfg80211_rx_unprot_mlme_mgmt [Tracepoint event] cfg80211:cfg80211_scan_done [Tracepoint event] cfg80211:cfg80211_sched_scan_results [Tracepoint event] cfg80211:cfg80211_sched_scan_stopped [Tracepoint event] cfg80211:cfg80211_send_assoc_timeout [Tracepoint event] cfg80211:cfg80211_send_auth_timeout [Tracepoint event] cfg80211:cfg80211_send_rx_assoc [Tracepoint event] cfg80211:cfg80211_send_rx_auth [Tracepoint event] cfg80211:cfg80211_stop_iface [Tracepoint event] cfg80211:cfg80211_tdls_oper_request [Tracepoint event] cfg80211:cfg80211_tx_mlme_mgmt [Tracepoint event] cfg80211:rdev_add_key [Tracepoint event] cfg80211:rdev_add_mpath [Tracepoint event] cfg80211:rdev_add_station [Tracepoint event] cfg80211:rdev_add_tx_ts [Tracepoint event] cfg80211:rdev_add_virtual_intf [Tracepoint event] cfg80211:rdev_assoc [Tracepoint event] cfg80211:rdev_auth [Tracepoint event] cfg80211:rdev_cancel_remain_on_channel [Tracepoint event] cfg80211:rdev_change_beacon [Tracepoint event] cfg80211:rdev_change_bss [Tracepoint event] cfg80211:rdev_change_mpath [Tracepoint event] cfg80211:rdev_change_station [Tracepoint event] cfg80211:rdev_change_virtual_intf [Tracepoint event] cfg80211:rdev_channel_switch [Tracepoint event] cfg80211:rdev_connect [Tracepoint event] cfg80211:rdev_crit_proto_start [Tracepoint event] cfg80211:rdev_crit_proto_stop [Tracepoint event] cfg80211:rdev_deauth [Tracepoint event] cfg80211:rdev_del_key [Tracepoint event] cfg80211:rdev_del_mpath [Tracepoint event] cfg80211:rdev_del_pmksa [Tracepoint event] cfg80211:rdev_del_station [Tracepoint event] cfg80211:rdev_del_tx_ts [Tracepoint event] cfg80211:rdev_del_virtual_intf [Tracepoint event] cfg80211:rdev_disassoc [Tracepoint event] cfg80211:rdev_disconnect [Tracepoint event] cfg80211:rdev_dump_mpath [Tracepoint event] cfg80211:rdev_dump_mpp [Tracepoint event] cfg80211:rdev_dump_station [Tracepoint event] cfg80211:rdev_dump_survey [Tracepoint event] cfg80211:rdev_flush_pmksa [Tracepoint event] cfg80211:rdev_get_antenna [Tracepoint event] cfg80211:rdev_get_channel [Tracepoint event] cfg80211:rdev_get_key [Tracepoint event] cfg80211:rdev_get_mesh_config [Tracepoint event] cfg80211:rdev_get_mpath [Tracepoint event] cfg80211:rdev_get_mpp [Tracepoint event] cfg80211:rdev_get_station [Tracepoint event] cfg80211:rdev_get_tx_power [Tracepoint event] cfg80211:rdev_join_ibss [Tracepoint event] cfg80211:rdev_join_mesh [Tracepoint event] cfg80211:rdev_join_ocb [Tracepoint event] cfg80211:rdev_leave_ibss [Tracepoint event] cfg80211:rdev_leave_mesh [Tracepoint event] cfg80211:rdev_leave_ocb [Tracepoint event] cfg80211:rdev_libertas_set_mesh_channel [Tracepoint event] cfg80211:rdev_mgmt_frame_register [Tracepoint event] cfg80211:rdev_mgmt_tx [Tracepoint event] cfg80211:rdev_mgmt_tx_cancel_wait [Tracepoint event] cfg80211:rdev_probe_client [Tracepoint event] cfg80211:rdev_remain_on_channel [Tracepoint event] cfg80211:rdev_resume [Tracepoint event] cfg80211:rdev_return_chandef [Tracepoint event] cfg80211:rdev_return_int [Tracepoint event] cfg80211:rdev_return_int_cookie [Tracepoint event] cfg80211:rdev_return_int_int [Tracepoint event] cfg80211:rdev_return_int_mesh_config [Tracepoint event] cfg80211:rdev_return_int_mpath_info [Tracepoint event] cfg80211:rdev_return_int_station_info [Tracepoint event] cfg80211:rdev_return_int_survey_info [Tracepoint event] cfg80211:rdev_return_int_tx_rx [Tracepoint event] cfg80211:rdev_return_void [Tracepoint event] cfg80211:rdev_return_void_tx_rx [Tracepoint event] cfg80211:rdev_return_wdev [Tracepoint event] cfg80211:rdev_rfkill_poll [Tracepoint event] cfg80211:rdev_scan [Tracepoint event] cfg80211:rdev_sched_scan_start [Tracepoint event] cfg80211:rdev_sched_scan_stop [Tracepoint event] cfg80211:rdev_set_antenna [Tracepoint event] cfg80211:rdev_set_ap_chanwidth [Tracepoint event] cfg80211:rdev_set_bitrate_mask [Tracepoint event] cfg80211:rdev_set_cqm_rssi_config [Tracepoint event] cfg80211:rdev_set_cqm_txe_config [Tracepoint event] cfg80211:rdev_set_default_key [Tracepoint event] cfg80211:rdev_set_default_mgmt_key [Tracepoint event] cfg80211:rdev_set_mac_acl [Tracepoint event] cfg80211:rdev_set_monitor_channel [Tracepoint event] cfg80211:rdev_set_noack_map [Tracepoint event] cfg80211:rdev_set_pmksa [Tracepoint event] cfg80211:rdev_set_power_mgmt [Tracepoint event] cfg80211:rdev_set_qos_map [Tracepoint event] cfg80211:rdev_set_rekey_data [Tracepoint event] cfg80211:rdev_set_tx_power [Tracepoint event] cfg80211:rdev_set_txq_params [Tracepoint event] cfg80211:rdev_set_wakeup [Tracepoint event] cfg80211:rdev_set_wds_peer [Tracepoint event] cfg80211:rdev_set_wiphy_params [Tracepoint event] cfg80211:rdev_start_ap [Tracepoint event] cfg80211:rdev_start_p2p_device [Tracepoint event] cfg80211:rdev_stop_ap [Tracepoint event] cfg80211:rdev_stop_p2p_device [Tracepoint event] cfg80211:rdev_suspend [Tracepoint event] cfg80211:rdev_tdls_cancel_channel_switch [Tracepoint event] cfg80211:rdev_tdls_channel_switch [Tracepoint event] cfg80211:rdev_tdls_mgmt [Tracepoint event] cfg80211:rdev_tdls_oper [Tracepoint event] cfg80211:rdev_update_ft_ies [Tracepoint event] cfg80211:rdev_update_mesh_config [Tracepoint event] clk:clk_disable [Tracepoint event] clk:clk_disable_complete [Tracepoint event] clk:clk_enable [Tracepoint event] clk:clk_enable_complete [Tracepoint event] clk:clk_prepare [Tracepoint event] clk:clk_prepare_complete [Tracepoint event] clk:clk_set_parent [Tracepoint event] clk:clk_set_parent_complete [Tracepoint event] clk:clk_set_phase [Tracepoint event] clk:clk_set_phase_complete [Tracepoint event] clk:clk_set_rate [Tracepoint event] clk:clk_set_rate_complete [Tracepoint event] clk:clk_unprepare [Tracepoint event] clk:clk_unprepare_complete [Tracepoint event] compaction:mm_compaction_begin [Tracepoint event] compaction:mm_compaction_defer_compaction [Tracepoint event] compaction:mm_compaction_defer_reset [Tracepoint event] compaction:mm_compaction_deferred [Tracepoint event] compaction:mm_compaction_end [Tracepoint event] compaction:mm_compaction_finished [Tracepoint event] compaction:mm_compaction_isolate_freepages [Tracepoint event] compaction:mm_compaction_isolate_migratepages [Tracepoint event] compaction:mm_compaction_migratepages [Tracepoint event] compaction:mm_compaction_suitable [Tracepoint event] compaction:mm_compaction_try_to_compact_pages [Tracepoint event] drm:drm_vblank_event [Tracepoint event] drm:drm_vblank_event_delivered [Tracepoint event] drm:drm_vblank_event_queued [Tracepoint event] exceptions:page_fault_kernel [Tracepoint event] exceptions:page_fault_user [Tracepoint event] ext4:ext4_alloc_da_blocks [Tracepoint event] ext4:ext4_allocate_blocks [Tracepoint event] ext4:ext4_allocate_inode [Tracepoint event] ext4:ext4_begin_ordered_truncate [Tracepoint event] ext4:ext4_collapse_range [Tracepoint event] ext4:ext4_da_release_space [Tracepoint event] ext4:ext4_da_reserve_space [Tracepoint event] ext4:ext4_da_update_reserve_space [Tracepoint event] ext4:ext4_da_write_begin [Tracepoint event] ext4:ext4_da_write_end [Tracepoint event] ext4:ext4_da_write_pages [Tracepoint event] ext4:ext4_da_write_pages_extent [Tracepoint event] ext4:ext4_direct_IO_enter [Tracepoint event] ext4:ext4_direct_IO_exit [Tracepoint event] ext4:ext4_discard_blocks [Tracepoint event] ext4:ext4_discard_preallocations [Tracepoint event] ext4:ext4_drop_inode [Tracepoint event] ext4:ext4_es_cache_extent [Tracepoint event] ext4:ext4_es_find_delayed_extent_range_enter [Tracepoint event] ext4:ext4_es_find_delayed_extent_range_exit [Tracepoint event] ext4:ext4_es_insert_extent [Tracepoint event] ext4:ext4_es_lookup_extent_enter [Tracepoint event] ext4:ext4_es_lookup_extent_exit [Tracepoint event] ext4:ext4_es_remove_extent [Tracepoint event] ext4:ext4_es_shrink [Tracepoint event] ext4:ext4_es_shrink_count [Tracepoint event] ext4:ext4_es_shrink_scan_enter [Tracepoint event] ext4:ext4_es_shrink_scan_exit [Tracepoint event] ext4:ext4_evict_inode [Tracepoint event] ext4:ext4_ext_convert_to_initialized_enter [Tracepoint event] ext4:ext4_ext_convert_to_initialized_fastpath [Tracepoint event] ext4:ext4_ext_handle_unwritten_extents [Tracepoint event] ext4:ext4_ext_in_cache [Tracepoint event] ext4:ext4_ext_load_extent [Tracepoint event] ext4:ext4_ext_map_blocks_enter [Tracepoint event] ext4:ext4_ext_map_blocks_exit [Tracepoint event] ext4:ext4_ext_put_in_cache [Tracepoint event] ext4:ext4_ext_remove_space [Tracepoint event] ext4:ext4_ext_remove_space_done [Tracepoint event] ext4:ext4_ext_rm_idx [Tracepoint event] ext4:ext4_ext_rm_leaf [Tracepoint event] ext4:ext4_ext_show_extent [Tracepoint event] ext4:ext4_fallocate_enter [Tracepoint event] ext4:ext4_fallocate_exit [Tracepoint event] ext4:ext4_find_delalloc_range [Tracepoint event] ext4:ext4_forget [Tracepoint event] ext4:ext4_free_blocks [Tracepoint event] ext4:ext4_free_inode [Tracepoint event] ext4:ext4_get_implied_cluster_alloc_exit [Tracepoint event] ext4:ext4_get_reserved_cluster_alloc [Tracepoint event] ext4:ext4_ind_map_blocks_enter [Tracepoint event] ext4:ext4_ind_map_blocks_exit [Tracepoint event] ext4:ext4_insert_range [Tracepoint event] ext4:ext4_invalidatepage [Tracepoint event] ext4:ext4_journal_start [Tracepoint event] ext4:ext4_journal_start_reserved [Tracepoint event] ext4:ext4_journalled_invalidatepage [Tracepoint event] ext4:ext4_journalled_write_end [Tracepoint event] ext4:ext4_load_inode [Tracepoint event] ext4:ext4_load_inode_bitmap [Tracepoint event] ext4:ext4_mark_inode_dirty [Tracepoint event] ext4:ext4_mb_bitmap_load [Tracepoint event] ext4:ext4_mb_buddy_bitmap_load [Tracepoint event] ext4:ext4_mb_discard_preallocations [Tracepoint event] ext4:ext4_mb_new_group_pa [Tracepoint event] ext4:ext4_mb_new_inode_pa [Tracepoint event] ext4:ext4_mb_release_group_pa [Tracepoint event] ext4:ext4_mb_release_inode_pa [Tracepoint event] ext4:ext4_mballoc_alloc [Tracepoint event] ext4:ext4_mballoc_discard [Tracepoint event] ext4:ext4_mballoc_free [Tracepoint event] ext4:ext4_mballoc_prealloc [Tracepoint event] ext4:ext4_other_inode_update_time [Tracepoint event] ext4:ext4_punch_hole [Tracepoint event] ext4:ext4_read_block_bitmap_load [Tracepoint event] ext4:ext4_readpage [Tracepoint event] ext4:ext4_releasepage [Tracepoint event] ext4:ext4_remove_blocks [Tracepoint event] ext4:ext4_request_blocks [Tracepoint event] ext4:ext4_request_inode [Tracepoint event] ext4:ext4_sync_file_enter [Tracepoint event] ext4:ext4_sync_file_exit [Tracepoint event] ext4:ext4_sync_fs [Tracepoint event] ext4:ext4_trim_all_free [Tracepoint event] ext4:ext4_trim_extent [Tracepoint event] ext4:ext4_truncate_enter [Tracepoint event] ext4:ext4_truncate_exit [Tracepoint event] ext4:ext4_unlink_enter [Tracepoint event] ext4:ext4_unlink_exit [Tracepoint event] ext4:ext4_write_begin [Tracepoint event] ext4:ext4_write_end [Tracepoint event] ext4:ext4_writepage [Tracepoint event] ext4:ext4_writepages [Tracepoint event] ext4:ext4_writepages_result [Tracepoint event] ext4:ext4_zero_range [Tracepoint event] fence:fence_annotate_wait_on [Tracepoint event] fence:fence_destroy [Tracepoint event] fence:fence_emit [Tracepoint event] fence:fence_enable_signal [Tracepoint event] fence:fence_init [Tracepoint event] fence:fence_signaled [Tracepoint event] fence:fence_wait_end [Tracepoint event] fence:fence_wait_start [Tracepoint event] filelock:break_lease_block [Tracepoint event] filelock:break_lease_noblock [Tracepoint event] filelock:break_lease_unblock [Tracepoint event] filelock:generic_add_lease [Tracepoint event] filelock:generic_delete_lease [Tracepoint event] filelock:time_out_leases [Tracepoint event] filemap:mm_filemap_add_to_page_cache [Tracepoint event] filemap:mm_filemap_delete_from_page_cache [Tracepoint event] ftrace:function [Tracepoint event] gpio:gpio_direction [Tracepoint event] gpio:gpio_value [Tracepoint event] hda:hda_get_response [Tracepoint event] hda:hda_send_cmd [Tracepoint event] hda:hda_unsol_event [Tracepoint event] hda:snd_hdac_stream_start [Tracepoint event] hda:snd_hdac_stream_stop [Tracepoint event] hda_controller:azx_get_position [Tracepoint event] hda_controller:azx_pcm_close [Tracepoint event] hda_controller:azx_pcm_hw_params [Tracepoint event] hda_controller:azx_pcm_open [Tracepoint event] hda_controller:azx_pcm_prepare [Tracepoint event] hda_controller:azx_pcm_trigger [Tracepoint event] hda_intel:azx_resume [Tracepoint event] hda_intel:azx_runtime_resume [Tracepoint event] hda_intel:azx_runtime_suspend [Tracepoint event] hda_intel:azx_suspend [Tracepoint event] i2c:i2c_read [Tracepoint event] i2c:i2c_reply [Tracepoint event] i2c:i2c_result [Tracepoint event] i2c:i2c_write [Tracepoint event] i2c:smbus_read [Tracepoint event] i2c:smbus_reply [Tracepoint event] i2c:smbus_result [Tracepoint event] i2c:smbus_write [Tracepoint event] iommu:add_device_to_group [Tracepoint event] iommu:attach_device_to_domain [Tracepoint event] iommu:detach_device_from_domain [Tracepoint event] iommu:io_page_fault [Tracepoint event] iommu:map [Tracepoint event] iommu:remove_device_from_group [Tracepoint event] iommu:unmap [Tracepoint event] irq:irq_handler_entry [Tracepoint event] irq:irq_handler_exit [Tracepoint event] irq:softirq_entry [Tracepoint event] irq:softirq_exit [Tracepoint event] irq:softirq_raise [Tracepoint event] irq_vectors:call_function_entry [Tracepoint event] irq_vectors:call_function_exit [Tracepoint event] irq_vectors:call_function_single_entry [Tracepoint event] irq_vectors:call_function_single_exit [Tracepoint event] irq_vectors:deferred_error_apic_entry [Tracepoint event] irq_vectors:deferred_error_apic_exit [Tracepoint event] irq_vectors:error_apic_entry [Tracepoint event] irq_vectors:error_apic_exit [Tracepoint event] irq_vectors:irq_work_entry [Tracepoint event] irq_vectors:irq_work_exit [Tracepoint event] irq_vectors:local_timer_entry [Tracepoint event] irq_vectors:local_timer_exit [Tracepoint event] irq_vectors:reschedule_entry [Tracepoint event] irq_vectors:reschedule_exit [Tracepoint event] irq_vectors:spurious_apic_entry [Tracepoint event] irq_vectors:spurious_apic_exit [Tracepoint event] irq_vectors:thermal_apic_entry [Tracepoint event] irq_vectors:thermal_apic_exit [Tracepoint event] irq_vectors:threshold_apic_entry [Tracepoint event] irq_vectors:threshold_apic_exit [Tracepoint event] irq_vectors:x86_platform_ipi_entry [Tracepoint event] irq_vectors:x86_platform_ipi_exit [Tracepoint event] jbd2:jbd2_checkpoint [Tracepoint event] jbd2:jbd2_checkpoint_stats [Tracepoint event] jbd2:jbd2_commit_flushing [Tracepoint event] jbd2:jbd2_commit_locking [Tracepoint event] jbd2:jbd2_commit_logging [Tracepoint event] jbd2:jbd2_drop_transaction [Tracepoint event] jbd2:jbd2_end_commit [Tracepoint event] jbd2:jbd2_handle_extend [Tracepoint event] jbd2:jbd2_handle_start [Tracepoint event] jbd2:jbd2_handle_stats [Tracepoint event] jbd2:jbd2_lock_buffer_stall [Tracepoint event] jbd2:jbd2_run_stats [Tracepoint event] jbd2:jbd2_start_commit [Tracepoint event] jbd2:jbd2_submit_inode_data [Tracepoint event] jbd2:jbd2_update_log_tail [Tracepoint event] jbd2:jbd2_write_superblock [Tracepoint event] kmem:kfree [Tracepoint event] kmem:kmalloc [Tracepoint event] kmem:kmalloc_node [Tracepoint event] kmem:kmem_cache_alloc [Tracepoint event] kmem:kmem_cache_alloc_node [Tracepoint event] kmem:kmem_cache_free [Tracepoint event] kmem:mm_page_alloc [Tracepoint event] kmem:mm_page_alloc_extfrag [Tracepoint event] kmem:mm_page_alloc_zone_locked [Tracepoint event] kmem:mm_page_free [Tracepoint event] kmem:mm_page_free_batched [Tracepoint event] kmem:mm_page_pcpu_drain [Tracepoint event] kvm:kvm_ack_irq [Tracepoint event] kvm:kvm_age_page [Tracepoint event] kvm:kvm_apic [Tracepoint event] kvm:kvm_apic_accept_irq [Tracepoint event] kvm:kvm_apic_ipi [Tracepoint event] kvm:kvm_async_pf_completed [Tracepoint event] kvm:kvm_async_pf_doublefault [Tracepoint event] kvm:kvm_async_pf_not_present [Tracepoint event] kvm:kvm_async_pf_ready [Tracepoint event] kvm:kvm_cpuid [Tracepoint event] kvm:kvm_cr [Tracepoint event] kvm:kvm_emulate_insn [Tracepoint event] kvm:kvm_enter_smm [Tracepoint event] kvm:kvm_entry [Tracepoint event] kvm:kvm_eoi [Tracepoint event] kvm:kvm_exit [Tracepoint event] kvm:kvm_fpu [Tracepoint event] kvm:kvm_hv_hypercall [Tracepoint event] kvm:kvm_hypercall [Tracepoint event] kvm:kvm_inj_exception [Tracepoint event] kvm:kvm_inj_virq [Tracepoint event] kvm:kvm_invlpga [Tracepoint event] kvm:kvm_ioapic_delayed_eoi_inj [Tracepoint event] kvm:kvm_ioapic_set_irq [Tracepoint event] kvm:kvm_mmio [Tracepoint event] kvm:kvm_msi_set_irq [Tracepoint event] kvm:kvm_msr [Tracepoint event] kvm:kvm_nested_intercepts [Tracepoint event] kvm:kvm_nested_intr_vmexit [Tracepoint event] kvm:kvm_nested_vmexit [Tracepoint event] kvm:kvm_nested_vmexit_inject [Tracepoint event] kvm:kvm_nested_vmrun [Tracepoint event] kvm:kvm_page_fault [Tracepoint event] kvm:kvm_pic_set_irq [Tracepoint event] kvm:kvm_pio [Tracepoint event] kvm:kvm_ple_window [Tracepoint event] kvm:kvm_pml_full [Tracepoint event] kvm:kvm_pv_eoi [Tracepoint event] kvm:kvm_pvclock_update [Tracepoint event] kvm:kvm_set_irq [Tracepoint event] kvm:kvm_skinit [Tracepoint event] kvm:kvm_track_tsc [Tracepoint event] kvm:kvm_try_async_get_page [Tracepoint event] kvm:kvm_update_master_clock [Tracepoint event] kvm:kvm_userspace_exit [Tracepoint event] kvm:kvm_vcpu_wakeup [Tracepoint event] kvm:kvm_wait_lapic_expire [Tracepoint event] kvm:kvm_write_tsc_offset [Tracepoint event] kvm:vcpu_match_mmio [Tracepoint event] kvmmmu:check_mmio_spte [Tracepoint event] kvmmmu:fast_page_fault [Tracepoint event] kvmmmu:handle_mmio_page_fault [Tracepoint event] kvmmmu:kvm_mmu_get_page [Tracepoint event] kvmmmu:kvm_mmu_invalidate_zap_all_pages [Tracepoint event] kvmmmu:kvm_mmu_pagetable_walk [Tracepoint event] kvmmmu:kvm_mmu_paging_element [Tracepoint event] kvmmmu:kvm_mmu_prepare_zap_page [Tracepoint event] kvmmmu:kvm_mmu_set_accessed_bit [Tracepoint event] kvmmmu:kvm_mmu_set_dirty_bit [Tracepoint event] kvmmmu:kvm_mmu_sync_page [Tracepoint event] kvmmmu:kvm_mmu_unsync_page [Tracepoint event] kvmmmu:kvm_mmu_walker_error [Tracepoint event] kvmmmu:mark_mmio_spte [Tracepoint event] libata:ata_eh_link_autopsy [Tracepoint event] libata:ata_eh_link_autopsy_qc [Tracepoint event] libata:ata_qc_complete_done [Tracepoint event] libata:ata_qc_complete_failed [Tracepoint event] libata:ata_qc_complete_internal [Tracepoint event] libata:ata_qc_issue [Tracepoint event] mce:mce_record [Tracepoint event] migrate:mm_migrate_pages [Tracepoint event] migrate:mm_numa_migrate_ratelimit [Tracepoint event] module:module_free [Tracepoint event] module:module_get [Tracepoint event] module:module_load [Tracepoint event] module:module_put [Tracepoint event] module:module_request [Tracepoint event] napi:napi_poll [Tracepoint event] net:napi_gro_frags_entry [Tracepoint event] net:napi_gro_receive_entry [Tracepoint event] net:net_dev_queue [Tracepoint event] net:net_dev_start_xmit [Tracepoint event] net:net_dev_xmit [Tracepoint event] net:netif_receive_skb [Tracepoint event] net:netif_receive_skb_entry [Tracepoint event] net:netif_rx [Tracepoint event] net:netif_rx_entry [Tracepoint event] net:netif_rx_ni_entry [Tracepoint event] nmi:nmi_handler [Tracepoint event] oom:oom_score_adj_update [Tracepoint event] pagemap:mm_lru_activate [Tracepoint event] pagemap:mm_lru_insertion [Tracepoint event] power:clock_disable [Tracepoint event] power:clock_enable [Tracepoint event] power:clock_set_rate [Tracepoint event] power:cpu_frequency [Tracepoint event] power:cpu_idle [Tracepoint event] power:dev_pm_qos_add_request [Tracepoint event] power:dev_pm_qos_remove_request [Tracepoint event] power:dev_pm_qos_update_request [Tracepoint event] power:device_pm_callback_end [Tracepoint event] power:device_pm_callback_start [Tracepoint event] power:pm_qos_add_request [Tracepoint event] power:pm_qos_remove_request [Tracepoint event] power:pm_qos_update_flags [Tracepoint event] power:pm_qos_update_request [Tracepoint event] power:pm_qos_update_request_timeout [Tracepoint event] power:pm_qos_update_target [Tracepoint event] power:power_domain_target [Tracepoint event] power:pstate_sample [Tracepoint event] power:suspend_resume [Tracepoint event] power:wakeup_source_activate [Tracepoint event] power:wakeup_source_deactivate [Tracepoint event] printk:console [Tracepoint event] radeon:radeon_bo_create [Tracepoint event] radeon:radeon_cs [Tracepoint event] radeon:radeon_fence_emit [Tracepoint event] radeon:radeon_fence_wait_begin [Tracepoint event] radeon:radeon_fence_wait_end [Tracepoint event] radeon:radeon_semaphore_signale [Tracepoint event] radeon:radeon_semaphore_wait [Tracepoint event] radeon:radeon_vm_bo_update [Tracepoint event] radeon:radeon_vm_flush [Tracepoint event] radeon:radeon_vm_grab_id [Tracepoint event] radeon:radeon_vm_set_page [Tracepoint event] random:add_device_randomness [Tracepoint event] random:add_disk_randomness [Tracepoint event] random:add_input_randomness [Tracepoint event] random:credit_entropy_bits [Tracepoint event] random:debit_entropy [Tracepoint event] random:extract_entropy [Tracepoint event] random:extract_entropy_user [Tracepoint event] random:get_random_bytes [Tracepoint event] random:get_random_bytes_arch [Tracepoint event] random:mix_pool_bytes [Tracepoint event] random:mix_pool_bytes_nolock [Tracepoint event] random:push_to_pool [Tracepoint event] random:random_read [Tracepoint event] random:urandom_read [Tracepoint event] random:xfer_secondary_pool [Tracepoint event] ras:aer_event [Tracepoint event] ras:extlog_mem_event [Tracepoint event] ras:mc_event [Tracepoint event] ras:memory_failure_event [Tracepoint event] raw_syscalls:sys_enter [Tracepoint event] raw_syscalls:sys_exit [Tracepoint event] rcu:rcu_utilization [Tracepoint event] regmap:regcache_drop_region [Tracepoint event] regmap:regcache_sync [Tracepoint event] regmap:regmap_async_complete_done [Tracepoint event] regmap:regmap_async_complete_start [Tracepoint event] regmap:regmap_async_io_complete [Tracepoint event] regmap:regmap_async_write_start [Tracepoint event] regmap:regmap_cache_bypass [Tracepoint event] regmap:regmap_cache_only [Tracepoint event] regmap:regmap_hw_read_done [Tracepoint event] regmap:regmap_hw_read_start [Tracepoint event] regmap:regmap_hw_write_done [Tracepoint event] regmap:regmap_hw_write_start [Tracepoint event] regmap:regmap_reg_read [Tracepoint event] regmap:regmap_reg_read_cache [Tracepoint event] regmap:regmap_reg_write [Tracepoint event] rpm:rpm_idle [Tracepoint event] rpm:rpm_resume [Tracepoint event] rpm:rpm_return_int [Tracepoint event] rpm:rpm_suspend [Tracepoint event] sched:sched_kthread_stop [Tracepoint event] sched:sched_kthread_stop_ret [Tracepoint event] sched:sched_migrate_task [Tracepoint event] sched:sched_move_numa [Tracepoint event] sched:sched_pi_setprio [Tracepoint event] sched:sched_process_exec [Tracepoint event] sched:sched_process_exit [Tracepoint event] sched:sched_process_fork [Tracepoint event] sched:sched_process_free [Tracepoint event] sched:sched_process_hang [Tracepoint event] sched:sched_process_wait [Tracepoint event] sched:sched_stat_blocked [Tracepoint event] sched:sched_stat_iowait [Tracepoint event] sched:sched_stat_runtime [Tracepoint event] sched:sched_stat_sleep [Tracepoint event] sched:sched_stat_wait [Tracepoint event] sched:sched_stick_numa [Tracepoint event] sched:sched_swap_numa [Tracepoint event] sched:sched_switch [Tracepoint event] sched:sched_wait_task [Tracepoint event] sched:sched_wake_idle_without_ipi [Tracepoint event] sched:sched_wakeup [Tracepoint event] sched:sched_wakeup_new [Tracepoint event] scsi:scsi_dispatch_cmd_done [Tracepoint event] scsi:scsi_dispatch_cmd_error [Tracepoint event] scsi:scsi_dispatch_cmd_start [Tracepoint event] scsi:scsi_dispatch_cmd_timeout [Tracepoint event] scsi:scsi_eh_wakeup [Tracepoint event] signal:signal_deliver [Tracepoint event] signal:signal_generate [Tracepoint event] skb:consume_skb [Tracepoint event] skb:kfree_skb [Tracepoint event] skb:skb_copy_datagram_iovec [Tracepoint event] sock:sock_exceed_buf_limit [Tracepoint event] sock:sock_rcvqueue_full [Tracepoint event] spi:spi_master_busy [Tracepoint event] spi:spi_master_idle [Tracepoint event] spi:spi_message_done [Tracepoint event] spi:spi_message_start [Tracepoint event] spi:spi_message_submit [Tracepoint event] spi:spi_transfer_start [Tracepoint event] spi:spi_transfer_stop [Tracepoint event] swiotlb:swiotlb_bounced [Tracepoint event] syscalls:sys_enter_accept [Tracepoint event] syscalls:sys_enter_accept4 [Tracepoint event] syscalls:sys_enter_access [Tracepoint event] syscalls:sys_enter_acct [Tracepoint event] syscalls:sys_enter_add_key [Tracepoint event] syscalls:sys_enter_adjtimex [Tracepoint event] syscalls:sys_enter_alarm [Tracepoint event] syscalls:sys_enter_bind [Tracepoint event] syscalls:sys_enter_brk [Tracepoint event] syscalls:sys_enter_capget [Tracepoint event] syscalls:sys_enter_capset [Tracepoint event] syscalls:sys_enter_chdir [Tracepoint event] syscalls:sys_enter_chmod [Tracepoint event] syscalls:sys_enter_chown [Tracepoint event] syscalls:sys_enter_chroot [Tracepoint event] syscalls:sys_enter_clock_adjtime [Tracepoint event] syscalls:sys_enter_clock_getres [Tracepoint event] syscalls:sys_enter_clock_gettime [Tracepoint event] syscalls:sys_enter_clock_nanosleep [Tracepoint event] syscalls:sys_enter_clock_settime [Tracepoint event] syscalls:sys_enter_close [Tracepoint event] syscalls:sys_enter_connect [Tracepoint event] syscalls:sys_enter_creat [Tracepoint event] syscalls:sys_enter_delete_module [Tracepoint event] syscalls:sys_enter_dup [Tracepoint event] syscalls:sys_enter_dup2 [Tracepoint event] syscalls:sys_enter_dup3 [Tracepoint event] syscalls:sys_enter_epoll_create [Tracepoint event] syscalls:sys_enter_epoll_create1 [Tracepoint event] syscalls:sys_enter_epoll_ctl [Tracepoint event] syscalls:sys_enter_epoll_pwait [Tracepoint event] syscalls:sys_enter_epoll_wait [Tracepoint event] syscalls:sys_enter_eventfd [Tracepoint event] syscalls:sys_enter_eventfd2 [Tracepoint event] syscalls:sys_enter_exit [Tracepoint event] syscalls:sys_enter_exit_group [Tracepoint event] syscalls:sys_enter_faccessat [Tracepoint event] syscalls:sys_enter_fadvise64 [Tracepoint event] syscalls:sys_enter_fallocate [Tracepoint event] syscalls:sys_enter_fanotify_init [Tracepoint event] syscalls:sys_enter_fanotify_mark [Tracepoint event] syscalls:sys_enter_fchdir [Tracepoint event] syscalls:sys_enter_fchmod [Tracepoint event] syscalls:sys_enter_fchmodat [Tracepoint event] syscalls:sys_enter_fchown [Tracepoint event] syscalls:sys_enter_fchownat [Tracepoint event] syscalls:sys_enter_fcntl [Tracepoint event] syscalls:sys_enter_fdatasync [Tracepoint event] syscalls:sys_enter_fgetxattr [Tracepoint event] syscalls:sys_enter_finit_module [Tracepoint event] syscalls:sys_enter_flistxattr [Tracepoint event] syscalls:sys_enter_flock [Tracepoint event] syscalls:sys_enter_fremovexattr [Tracepoint event] syscalls:sys_enter_fsetxattr [Tracepoint event] syscalls:sys_enter_fstatfs [Tracepoint event] syscalls:sys_enter_fsync [Tracepoint event] syscalls:sys_enter_ftruncate [Tracepoint event] syscalls:sys_enter_futex [Tracepoint event] syscalls:sys_enter_futimesat [Tracepoint event] syscalls:sys_enter_get_mempolicy [Tracepoint event] syscalls:sys_enter_get_robust_list [Tracepoint event] syscalls:sys_enter_getcpu [Tracepoint event] syscalls:sys_enter_getcwd [Tracepoint event] syscalls:sys_enter_getdents [Tracepoint event] syscalls:sys_enter_getdents64 [Tracepoint event] syscalls:sys_enter_getegid [Tracepoint event] syscalls:sys_enter_geteuid [Tracepoint event] syscalls:sys_enter_getgid [Tracepoint event] syscalls:sys_enter_getgroups [Tracepoint event] syscalls:sys_enter_getitimer [Tracepoint event] syscalls:sys_enter_getpeername [Tracepoint event] syscalls:sys_enter_getpgid [Tracepoint event] syscalls:sys_enter_getpgrp [Tracepoint event] syscalls:sys_enter_getpid [Tracepoint event] syscalls:sys_enter_getppid [Tracepoint event] syscalls:sys_enter_getpriority [Tracepoint event] syscalls:sys_enter_getrandom [Tracepoint event] syscalls:sys_enter_getresgid [Tracepoint event] syscalls:sys_enter_getresuid [Tracepoint event] syscalls:sys_enter_getrlimit [Tracepoint event] syscalls:sys_enter_getrusage [Tracepoint event] syscalls:sys_enter_getsid [Tracepoint event] syscalls:sys_enter_getsockname [Tracepoint event] syscalls:sys_enter_getsockopt [Tracepoint event] syscalls:sys_enter_gettid [Tracepoint event] syscalls:sys_enter_gettimeofday [Tracepoint event] syscalls:sys_enter_getuid [Tracepoint event] syscalls:sys_enter_getxattr [Tracepoint event] syscalls:sys_enter_init_module [Tracepoint event] syscalls:sys_enter_inotify_add_watch [Tracepoint event] syscalls:sys_enter_inotify_init [Tracepoint event] syscalls:sys_enter_inotify_init1 [Tracepoint event] syscalls:sys_enter_inotify_rm_watch [Tracepoint event] syscalls:sys_enter_io_cancel [Tracepoint event] syscalls:sys_enter_io_destroy [Tracepoint event] syscalls:sys_enter_io_getevents [Tracepoint event] syscalls:sys_enter_io_setup [Tracepoint event] syscalls:sys_enter_io_submit [Tracepoint event] syscalls:sys_enter_ioctl [Tracepoint event] syscalls:sys_enter_iopl [Tracepoint event] syscalls:sys_enter_ioprio_get [Tracepoint event] syscalls:sys_enter_ioprio_set [Tracepoint event] syscalls:sys_enter_kcmp [Tracepoint event] syscalls:sys_enter_kexec_load [Tracepoint event] syscalls:sys_enter_keyctl [Tracepoint event] syscalls:sys_enter_kill [Tracepoint event] syscalls:sys_enter_lchown [Tracepoint event] syscalls:sys_enter_lgetxattr [Tracepoint event] syscalls:sys_enter_link [Tracepoint event] syscalls:sys_enter_linkat [Tracepoint event] syscalls:sys_enter_listen [Tracepoint event] syscalls:sys_enter_listxattr [Tracepoint event] syscalls:sys_enter_llistxattr [Tracepoint event] syscalls:sys_enter_lookup_dcookie [Tracepoint event] syscalls:sys_enter_lremovexattr [Tracepoint event] syscalls:sys_enter_lseek [Tracepoint event] syscalls:sys_enter_lsetxattr [Tracepoint event] syscalls:sys_enter_madvise [Tracepoint event] syscalls:sys_enter_mbind [Tracepoint event] syscalls:sys_enter_memfd_create [Tracepoint event] syscalls:sys_enter_migrate_pages [Tracepoint event] syscalls:sys_enter_mincore [Tracepoint event] syscalls:sys_enter_mkdir [Tracepoint event] syscalls:sys_enter_mkdirat [Tracepoint event] syscalls:sys_enter_mknod [Tracepoint event] syscalls:sys_enter_mknodat [Tracepoint event] syscalls:sys_enter_mlock [Tracepoint event] syscalls:sys_enter_mlockall [Tracepoint event] syscalls:sys_enter_mmap [Tracepoint event] syscalls:sys_enter_mount [Tracepoint event] syscalls:sys_enter_move_pages [Tracepoint event] syscalls:sys_enter_mprotect [Tracepoint event] syscalls:sys_enter_mq_getsetattr [Tracepoint event] syscalls:sys_enter_mq_notify [Tracepoint event] syscalls:sys_enter_mq_open [Tracepoint event] syscalls:sys_enter_mq_timedreceive [Tracepoint event] syscalls:sys_enter_mq_timedsend [Tracepoint event] syscalls:sys_enter_mq_unlink [Tracepoint event] syscalls:sys_enter_mremap [Tracepoint event] syscalls:sys_enter_msgctl [Tracepoint event] syscalls:sys_enter_msgget [Tracepoint event] syscalls:sys_enter_msgrcv [Tracepoint event] syscalls:sys_enter_msgsnd [Tracepoint event] syscalls:sys_enter_msync [Tracepoint event] syscalls:sys_enter_munlock [Tracepoint event] syscalls:sys_enter_munlockall [Tracepoint event] syscalls:sys_enter_munmap [Tracepoint event] syscalls:sys_enter_name_to_handle_at [Tracepoint event] syscalls:sys_enter_nanosleep [Tracepoint event] syscalls:sys_enter_newfstat [Tracepoint event] syscalls:sys_enter_newfstatat [Tracepoint event] syscalls:sys_enter_newlstat [Tracepoint event] syscalls:sys_enter_newstat [Tracepoint event] syscalls:sys_enter_newuname [Tracepoint event] syscalls:sys_enter_open [Tracepoint event] syscalls:sys_enter_open_by_handle_at [Tracepoint event] syscalls:sys_enter_openat [Tracepoint event] syscalls:sys_enter_pause [Tracepoint event] syscalls:sys_enter_perf_event_open [Tracepoint event] syscalls:sys_enter_personality [Tracepoint event] syscalls:sys_enter_pipe [Tracepoint event] syscalls:sys_enter_pipe2 [Tracepoint event] syscalls:sys_enter_pivot_root [Tracepoint event] syscalls:sys_enter_poll [Tracepoint event] syscalls:sys_enter_ppoll [Tracepoint event] syscalls:sys_enter_prctl [Tracepoint event] syscalls:sys_enter_pread64 [Tracepoint event] syscalls:sys_enter_preadv [Tracepoint event] syscalls:sys_enter_prlimit64 [Tracepoint event] syscalls:sys_enter_process_vm_readv [Tracepoint event] syscalls:sys_enter_process_vm_writev [Tracepoint event] syscalls:sys_enter_pselect6 [Tracepoint event] syscalls:sys_enter_ptrace [Tracepoint event] syscalls:sys_enter_pwrite64 [Tracepoint event] syscalls:sys_enter_pwritev [Tracepoint event] syscalls:sys_enter_quotactl [Tracepoint event] syscalls:sys_enter_read [Tracepoint event] syscalls:sys_enter_readahead [Tracepoint event] syscalls:sys_enter_readlink [Tracepoint event] syscalls:sys_enter_readlinkat [Tracepoint event] syscalls:sys_enter_readv [Tracepoint event] syscalls:sys_enter_reboot [Tracepoint event] syscalls:sys_enter_recvfrom [Tracepoint event] syscalls:sys_enter_recvmmsg [Tracepoint event] syscalls:sys_enter_recvmsg [Tracepoint event] syscalls:sys_enter_remap_file_pages [Tracepoint event] syscalls:sys_enter_removexattr [Tracepoint event] syscalls:sys_enter_rename [Tracepoint event] syscalls:sys_enter_renameat [Tracepoint event] syscalls:sys_enter_renameat2 [Tracepoint event] syscalls:sys_enter_request_key [Tracepoint event] syscalls:sys_enter_restart_syscall [Tracepoint event] syscalls:sys_enter_rmdir [Tracepoint event] syscalls:sys_enter_rt_sigaction [Tracepoint event] syscalls:sys_enter_rt_sigpending [Tracepoint event] syscalls:sys_enter_rt_sigprocmask [Tracepoint event] syscalls:sys_enter_rt_sigqueueinfo [Tracepoint event] syscalls:sys_enter_rt_sigsuspend [Tracepoint event] syscalls:sys_enter_rt_sigtimedwait [Tracepoint event] syscalls:sys_enter_rt_tgsigqueueinfo [Tracepoint event] syscalls:sys_enter_sched_get_priority_max [Tracepoint event] syscalls:sys_enter_sched_get_priority_min [Tracepoint event] syscalls:sys_enter_sched_getaffinity [Tracepoint event] syscalls:sys_enter_sched_getattr [Tracepoint event] syscalls:sys_enter_sched_getparam [Tracepoint event] syscalls:sys_enter_sched_getscheduler [Tracepoint event] syscalls:sys_enter_sched_rr_get_interval [Tracepoint event] syscalls:sys_enter_sched_setaffinity [Tracepoint event] syscalls:sys_enter_sched_setattr [Tracepoint event] syscalls:sys_enter_sched_setparam [Tracepoint event] syscalls:sys_enter_sched_setscheduler [Tracepoint event] syscalls:sys_enter_sched_yield [Tracepoint event] syscalls:sys_enter_seccomp [Tracepoint event] syscalls:sys_enter_select [Tracepoint event] syscalls:sys_enter_semctl [Tracepoint event] syscalls:sys_enter_semget [Tracepoint event] syscalls:sys_enter_semop [Tracepoint event] syscalls:sys_enter_semtimedop [Tracepoint event] syscalls:sys_enter_sendfile64 [Tracepoint event] syscalls:sys_enter_sendmmsg [Tracepoint event] syscalls:sys_enter_sendmsg [Tracepoint event] syscalls:sys_enter_sendto [Tracepoint event] syscalls:sys_enter_set_mempolicy [Tracepoint event] syscalls:sys_enter_set_robust_list [Tracepoint event] syscalls:sys_enter_set_tid_address [Tracepoint event] syscalls:sys_enter_setdomainname [Tracepoint event] syscalls:sys_enter_setfsgid [Tracepoint event] syscalls:sys_enter_setfsuid [Tracepoint event] syscalls:sys_enter_setgid [Tracepoint event] syscalls:sys_enter_setgroups [Tracepoint event] syscalls:sys_enter_sethostname [Tracepoint event] syscalls:sys_enter_setitimer [Tracepoint event] syscalls:sys_enter_setns [Tracepoint event] syscalls:sys_enter_setpgid [Tracepoint event] syscalls:sys_enter_setpriority [Tracepoint event] syscalls:sys_enter_setregid [Tracepoint event] syscalls:sys_enter_setresgid [Tracepoint event] syscalls:sys_enter_setresuid [Tracepoint event] syscalls:sys_enter_setreuid [Tracepoint event] syscalls:sys_enter_setrlimit [Tracepoint event] syscalls:sys_enter_setsid [Tracepoint event] syscalls:sys_enter_setsockopt [Tracepoint event] syscalls:sys_enter_settimeofday [Tracepoint event] syscalls:sys_enter_setuid [Tracepoint event] syscalls:sys_enter_setxattr [Tracepoint event] syscalls:sys_enter_shmat [Tracepoint event] syscalls:sys_enter_shmctl [Tracepoint event] syscalls:sys_enter_shmdt [Tracepoint event] syscalls:sys_enter_shmget [Tracepoint event] syscalls:sys_enter_shutdown [Tracepoint event] syscalls:sys_enter_sigaltstack [Tracepoint event] syscalls:sys_enter_signalfd [Tracepoint event] syscalls:sys_enter_signalfd4 [Tracepoint event] syscalls:sys_enter_socket [Tracepoint event] syscalls:sys_enter_socketpair [Tracepoint event] syscalls:sys_enter_splice [Tracepoint event] syscalls:sys_enter_statfs [Tracepoint event] syscalls:sys_enter_swapoff [Tracepoint event] syscalls:sys_enter_swapon [Tracepoint event] syscalls:sys_enter_symlink [Tracepoint event] syscalls:sys_enter_symlinkat [Tracepoint event] syscalls:sys_enter_sync [Tracepoint event] syscalls:sys_enter_sync_file_range [Tracepoint event] syscalls:sys_enter_syncfs [Tracepoint event] syscalls:sys_enter_sysctl [Tracepoint event] syscalls:sys_enter_sysfs [Tracepoint event] syscalls:sys_enter_sysinfo [Tracepoint event] syscalls:sys_enter_syslog [Tracepoint event] syscalls:sys_enter_tee [Tracepoint event] syscalls:sys_enter_tgkill [Tracepoint event] syscalls:sys_enter_time [Tracepoint event] syscalls:sys_enter_timer_create [Tracepoint event] syscalls:sys_enter_timer_delete [Tracepoint event] syscalls:sys_enter_timer_getoverrun [Tracepoint event] syscalls:sys_enter_timer_gettime [Tracepoint event] syscalls:sys_enter_timer_settime [Tracepoint event] syscalls:sys_enter_timerfd_create [Tracepoint event] syscalls:sys_enter_timerfd_gettime [Tracepoint event] syscalls:sys_enter_timerfd_settime [Tracepoint event] syscalls:sys_enter_times [Tracepoint event] syscalls:sys_enter_tkill [Tracepoint event] syscalls:sys_enter_truncate [Tracepoint event] syscalls:sys_enter_umask [Tracepoint event] syscalls:sys_enter_umount [Tracepoint event] syscalls:sys_enter_unlink [Tracepoint event] syscalls:sys_enter_unlinkat [Tracepoint event] syscalls:sys_enter_unshare [Tracepoint event] syscalls:sys_enter_ustat [Tracepoint event] syscalls:sys_enter_utime [Tracepoint event] syscalls:sys_enter_utimensat [Tracepoint event] syscalls:sys_enter_utimes [Tracepoint event] syscalls:sys_enter_vhangup [Tracepoint event] syscalls:sys_enter_vmsplice [Tracepoint event] syscalls:sys_enter_wait4 [Tracepoint event] syscalls:sys_enter_waitid [Tracepoint event] syscalls:sys_enter_write [Tracepoint event] syscalls:sys_enter_writev [Tracepoint event] syscalls:sys_exit_accept [Tracepoint event] syscalls:sys_exit_accept4 [Tracepoint event] syscalls:sys_exit_access [Tracepoint event] syscalls:sys_exit_acct [Tracepoint event] syscalls:sys_exit_add_key [Tracepoint event] syscalls:sys_exit_adjtimex [Tracepoint event] syscalls:sys_exit_alarm [Tracepoint event] syscalls:sys_exit_bind [Tracepoint event] syscalls:sys_exit_brk [Tracepoint event] syscalls:sys_exit_capget [Tracepoint event] syscalls:sys_exit_capset [Tracepoint event] syscalls:sys_exit_chdir [Tracepoint event] syscalls:sys_exit_chmod [Tracepoint event] syscalls:sys_exit_chown [Tracepoint event] syscalls:sys_exit_chroot [Tracepoint event] syscalls:sys_exit_clock_adjtime [Tracepoint event] syscalls:sys_exit_clock_getres [Tracepoint event] syscalls:sys_exit_clock_gettime [Tracepoint event] syscalls:sys_exit_clock_nanosleep [Tracepoint event] syscalls:sys_exit_clock_settime [Tracepoint event] syscalls:sys_exit_close [Tracepoint event] syscalls:sys_exit_connect [Tracepoint event] syscalls:sys_exit_creat [Tracepoint event] syscalls:sys_exit_delete_module [Tracepoint event] syscalls:sys_exit_dup [Tracepoint event] syscalls:sys_exit_dup2 [Tracepoint event] syscalls:sys_exit_dup3 [Tracepoint event] syscalls:sys_exit_epoll_create [Tracepoint event] syscalls:sys_exit_epoll_create1 [Tracepoint event] syscalls:sys_exit_epoll_ctl [Tracepoint event] syscalls:sys_exit_epoll_pwait [Tracepoint event] syscalls:sys_exit_epoll_wait [Tracepoint event] syscalls:sys_exit_eventfd [Tracepoint event] syscalls:sys_exit_eventfd2 [Tracepoint event] syscalls:sys_exit_exit [Tracepoint event] syscalls:sys_exit_exit_group [Tracepoint event] syscalls:sys_exit_faccessat [Tracepoint event] syscalls:sys_exit_fadvise64 [Tracepoint event] syscalls:sys_exit_fallocate [Tracepoint event] syscalls:sys_exit_fanotify_init [Tracepoint event] syscalls:sys_exit_fanotify_mark [Tracepoint event] syscalls:sys_exit_fchdir [Tracepoint event] syscalls:sys_exit_fchmod [Tracepoint event] syscalls:sys_exit_fchmodat [Tracepoint event] syscalls:sys_exit_fchown [Tracepoint event] syscalls:sys_exit_fchownat [Tracepoint event] syscalls:sys_exit_fcntl [Tracepoint event] syscalls:sys_exit_fdatasync [Tracepoint event] syscalls:sys_exit_fgetxattr [Tracepoint event] syscalls:sys_exit_finit_module [Tracepoint event] syscalls:sys_exit_flistxattr [Tracepoint event] syscalls:sys_exit_flock [Tracepoint event] syscalls:sys_exit_fremovexattr [Tracepoint event] syscalls:sys_exit_fsetxattr [Tracepoint event] syscalls:sys_exit_fstatfs [Tracepoint event] syscalls:sys_exit_fsync [Tracepoint event] syscalls:sys_exit_ftruncate [Tracepoint event] syscalls:sys_exit_futex [Tracepoint event] syscalls:sys_exit_futimesat [Tracepoint event] syscalls:sys_exit_get_mempolicy [Tracepoint event] syscalls:sys_exit_get_robust_list [Tracepoint event] syscalls:sys_exit_getcpu [Tracepoint event] syscalls:sys_exit_getcwd [Tracepoint event] syscalls:sys_exit_getdents [Tracepoint event] syscalls:sys_exit_getdents64 [Tracepoint event] syscalls:sys_exit_getegid [Tracepoint event] syscalls:sys_exit_geteuid [Tracepoint event] syscalls:sys_exit_getgid [Tracepoint event] syscalls:sys_exit_getgroups [Tracepoint event] syscalls:sys_exit_getitimer [Tracepoint event] syscalls:sys_exit_getpeername [Tracepoint event] syscalls:sys_exit_getpgid [Tracepoint event] syscalls:sys_exit_getpgrp [Tracepoint event] syscalls:sys_exit_getpid [Tracepoint event] syscalls:sys_exit_getppid [Tracepoint event] syscalls:sys_exit_getpriority [Tracepoint event] syscalls:sys_exit_getrandom [Tracepoint event] syscalls:sys_exit_getresgid [Tracepoint event] syscalls:sys_exit_getresuid [Tracepoint event] syscalls:sys_exit_getrlimit [Tracepoint event] syscalls:sys_exit_getrusage [Tracepoint event] syscalls:sys_exit_getsid [Tracepoint event] syscalls:sys_exit_getsockname [Tracepoint event] syscalls:sys_exit_getsockopt [Tracepoint event] syscalls:sys_exit_gettid [Tracepoint event] syscalls:sys_exit_gettimeofday [Tracepoint event] syscalls:sys_exit_getuid [Tracepoint event] syscalls:sys_exit_getxattr [Tracepoint event] syscalls:sys_exit_init_module [Tracepoint event] syscalls:sys_exit_inotify_add_watch [Tracepoint event] syscalls:sys_exit_inotify_init [Tracepoint event] syscalls:sys_exit_inotify_init1 [Tracepoint event] syscalls:sys_exit_inotify_rm_watch [Tracepoint event] syscalls:sys_exit_io_cancel [Tracepoint event] syscalls:sys_exit_io_destroy [Tracepoint event] syscalls:sys_exit_io_getevents [Tracepoint event] syscalls:sys_exit_io_setup [Tracepoint event] syscalls:sys_exit_io_submit [Tracepoint event] syscalls:sys_exit_ioctl [Tracepoint event] syscalls:sys_exit_iopl [Tracepoint event] syscalls:sys_exit_ioprio_get [Tracepoint event] syscalls:sys_exit_ioprio_set [Tracepoint event] syscalls:sys_exit_kcmp [Tracepoint event] syscalls:sys_exit_kexec_load [Tracepoint event] syscalls:sys_exit_keyctl [Tracepoint event] syscalls:sys_exit_kill [Tracepoint event] syscalls:sys_exit_lchown [Tracepoint event] syscalls:sys_exit_lgetxattr [Tracepoint event] syscalls:sys_exit_link [Tracepoint event] syscalls:sys_exit_linkat [Tracepoint event] syscalls:sys_exit_listen [Tracepoint event] syscalls:sys_exit_listxattr [Tracepoint event] syscalls:sys_exit_llistxattr [Tracepoint event] syscalls:sys_exit_lookup_dcookie [Tracepoint event] syscalls:sys_exit_lremovexattr [Tracepoint event] syscalls:sys_exit_lseek [Tracepoint event] syscalls:sys_exit_lsetxattr [Tracepoint event] syscalls:sys_exit_madvise [Tracepoint event] syscalls:sys_exit_mbind [Tracepoint event] syscalls:sys_exit_memfd_create [Tracepoint event] syscalls:sys_exit_migrate_pages [Tracepoint event] syscalls:sys_exit_mincore [Tracepoint event] syscalls:sys_exit_mkdir [Tracepoint event] syscalls:sys_exit_mkdirat [Tracepoint event] syscalls:sys_exit_mknod [Tracepoint event] syscalls:sys_exit_mknodat [Tracepoint event] syscalls:sys_exit_mlock [Tracepoint event] syscalls:sys_exit_mlockall [Tracepoint event] syscalls:sys_exit_mmap [Tracepoint event] syscalls:sys_exit_mount [Tracepoint event] syscalls:sys_exit_move_pages [Tracepoint event] syscalls:sys_exit_mprotect [Tracepoint event] syscalls:sys_exit_mq_getsetattr [Tracepoint event] syscalls:sys_exit_mq_notify [Tracepoint event] syscalls:sys_exit_mq_open [Tracepoint event] syscalls:sys_exit_mq_timedreceive [Tracepoint event] syscalls:sys_exit_mq_timedsend [Tracepoint event] syscalls:sys_exit_mq_unlink [Tracepoint event] syscalls:sys_exit_mremap [Tracepoint event] syscalls:sys_exit_msgctl [Tracepoint event] syscalls:sys_exit_msgget [Tracepoint event] syscalls:sys_exit_msgrcv [Tracepoint event] syscalls:sys_exit_msgsnd [Tracepoint event] syscalls:sys_exit_msync [Tracepoint event] syscalls:sys_exit_munlock [Tracepoint event] syscalls:sys_exit_munlockall [Tracepoint event] syscalls:sys_exit_munmap [Tracepoint event] syscalls:sys_exit_name_to_handle_at [Tracepoint event] syscalls:sys_exit_nanosleep [Tracepoint event] syscalls:sys_exit_newfstat [Tracepoint event] syscalls:sys_exit_newfstatat [Tracepoint event] syscalls:sys_exit_newlstat [Tracepoint event] syscalls:sys_exit_newstat [Tracepoint event] syscalls:sys_exit_newuname [Tracepoint event] syscalls:sys_exit_open [Tracepoint event] syscalls:sys_exit_open_by_handle_at [Tracepoint event] syscalls:sys_exit_openat [Tracepoint event] syscalls:sys_exit_pause [Tracepoint event] syscalls:sys_exit_perf_event_open [Tracepoint event] syscalls:sys_exit_personality [Tracepoint event] syscalls:sys_exit_pipe [Tracepoint event] syscalls:sys_exit_pipe2 [Tracepoint event] syscalls:sys_exit_pivot_root [Tracepoint event] syscalls:sys_exit_poll [Tracepoint event] syscalls:sys_exit_ppoll [Tracepoint event] syscalls:sys_exit_prctl [Tracepoint event] syscalls:sys_exit_pread64 [Tracepoint event] syscalls:sys_exit_preadv [Tracepoint event] syscalls:sys_exit_prlimit64 [Tracepoint event] syscalls:sys_exit_process_vm_readv [Tracepoint event] syscalls:sys_exit_process_vm_writev [Tracepoint event] syscalls:sys_exit_pselect6 [Tracepoint event] syscalls:sys_exit_ptrace [Tracepoint event] syscalls:sys_exit_pwrite64 [Tracepoint event] syscalls:sys_exit_pwritev [Tracepoint event] syscalls:sys_exit_quotactl [Tracepoint event] syscalls:sys_exit_read [Tracepoint event] syscalls:sys_exit_readahead [Tracepoint event] syscalls:sys_exit_readlink [Tracepoint event] syscalls:sys_exit_readlinkat [Tracepoint event] syscalls:sys_exit_readv [Tracepoint event] syscalls:sys_exit_reboot [Tracepoint event] syscalls:sys_exit_recvfrom [Tracepoint event] syscalls:sys_exit_recvmmsg [Tracepoint event] syscalls:sys_exit_recvmsg [Tracepoint event] syscalls:sys_exit_remap_file_pages [Tracepoint event] syscalls:sys_exit_removexattr [Tracepoint event] syscalls:sys_exit_rename [Tracepoint event] syscalls:sys_exit_renameat [Tracepoint event] syscalls:sys_exit_renameat2 [Tracepoint event] syscalls:sys_exit_request_key [Tracepoint event] syscalls:sys_exit_restart_syscall [Tracepoint event] syscalls:sys_exit_rmdir [Tracepoint event] syscalls:sys_exit_rt_sigaction [Tracepoint event] syscalls:sys_exit_rt_sigpending [Tracepoint event] syscalls:sys_exit_rt_sigprocmask [Tracepoint event] syscalls:sys_exit_rt_sigqueueinfo [Tracepoint event] syscalls:sys_exit_rt_sigsuspend [Tracepoint event] syscalls:sys_exit_rt_sigtimedwait [Tracepoint event] syscalls:sys_exit_rt_tgsigqueueinfo [Tracepoint event] syscalls:sys_exit_sched_get_priority_max [Tracepoint event] syscalls:sys_exit_sched_get_priority_min [Tracepoint event] syscalls:sys_exit_sched_getaffinity [Tracepoint event] syscalls:sys_exit_sched_getattr [Tracepoint event] syscalls:sys_exit_sched_getparam [Tracepoint event] syscalls:sys_exit_sched_getscheduler [Tracepoint event] syscalls:sys_exit_sched_rr_get_interval [Tracepoint event] syscalls:sys_exit_sched_setaffinity [Tracepoint event] syscalls:sys_exit_sched_setattr [Tracepoint event] syscalls:sys_exit_sched_setparam [Tracepoint event] syscalls:sys_exit_sched_setscheduler [Tracepoint event] syscalls:sys_exit_sched_yield [Tracepoint event] syscalls:sys_exit_seccomp [Tracepoint event] syscalls:sys_exit_select [Tracepoint event] syscalls:sys_exit_semctl [Tracepoint event] syscalls:sys_exit_semget [Tracepoint event] syscalls:sys_exit_semop [Tracepoint event] syscalls:sys_exit_semtimedop [Tracepoint event] syscalls:sys_exit_sendfile64 [Tracepoint event] syscalls:sys_exit_sendmmsg [Tracepoint event] syscalls:sys_exit_sendmsg [Tracepoint event] syscalls:sys_exit_sendto [Tracepoint event] syscalls:sys_exit_set_mempolicy [Tracepoint event] syscalls:sys_exit_set_robust_list [Tracepoint event] syscalls:sys_exit_set_tid_address [Tracepoint event] syscalls:sys_exit_setdomainname [Tracepoint event] syscalls:sys_exit_setfsgid [Tracepoint event] syscalls:sys_exit_setfsuid [Tracepoint event] syscalls:sys_exit_setgid [Tracepoint event] syscalls:sys_exit_setgroups [Tracepoint event] syscalls:sys_exit_sethostname [Tracepoint event] syscalls:sys_exit_setitimer [Tracepoint event] syscalls:sys_exit_setns [Tracepoint event] syscalls:sys_exit_setpgid [Tracepoint event] syscalls:sys_exit_setpriority [Tracepoint event] syscalls:sys_exit_setregid [Tracepoint event] syscalls:sys_exit_setresgid [Tracepoint event] syscalls:sys_exit_setresuid [Tracepoint event] syscalls:sys_exit_setreuid [Tracepoint event] syscalls:sys_exit_setrlimit [Tracepoint event] syscalls:sys_exit_setsid [Tracepoint event] syscalls:sys_exit_setsockopt [Tracepoint event] syscalls:sys_exit_settimeofday [Tracepoint event] syscalls:sys_exit_setuid [Tracepoint event] syscalls:sys_exit_setxattr [Tracepoint event] syscalls:sys_exit_shmat [Tracepoint event] syscalls:sys_exit_shmctl [Tracepoint event] syscalls:sys_exit_shmdt [Tracepoint event] syscalls:sys_exit_shmget [Tracepoint event] syscalls:sys_exit_shutdown [Tracepoint event] syscalls:sys_exit_sigaltstack [Tracepoint event] syscalls:sys_exit_signalfd [Tracepoint event] syscalls:sys_exit_signalfd4 [Tracepoint event] syscalls:sys_exit_socket [Tracepoint event] syscalls:sys_exit_socketpair [Tracepoint event] syscalls:sys_exit_splice [Tracepoint event] syscalls:sys_exit_statfs [Tracepoint event] syscalls:sys_exit_swapoff [Tracepoint event] syscalls:sys_exit_swapon [Tracepoint event] syscalls:sys_exit_symlink [Tracepoint event] syscalls:sys_exit_symlinkat [Tracepoint event] syscalls:sys_exit_sync [Tracepoint event] syscalls:sys_exit_sync_file_range [Tracepoint event] syscalls:sys_exit_syncfs [Tracepoint event] syscalls:sys_exit_sysctl [Tracepoint event] syscalls:sys_exit_sysfs [Tracepoint event] syscalls:sys_exit_sysinfo [Tracepoint event] syscalls:sys_exit_syslog [Tracepoint event] syscalls:sys_exit_tee [Tracepoint event] syscalls:sys_exit_tgkill [Tracepoint event] syscalls:sys_exit_time [Tracepoint event] syscalls:sys_exit_timer_create [Tracepoint event] syscalls:sys_exit_timer_delete [Tracepoint event] syscalls:sys_exit_timer_getoverrun [Tracepoint event] syscalls:sys_exit_timer_gettime [Tracepoint event] syscalls:sys_exit_timer_settime [Tracepoint event] syscalls:sys_exit_timerfd_create [Tracepoint event] syscalls:sys_exit_timerfd_gettime [Tracepoint event] syscalls:sys_exit_timerfd_settime [Tracepoint event] syscalls:sys_exit_times [Tracepoint event] syscalls:sys_exit_tkill [Tracepoint event] syscalls:sys_exit_truncate [Tracepoint event] syscalls:sys_exit_umask [Tracepoint event] syscalls:sys_exit_umount [Tracepoint event] syscalls:sys_exit_unlink [Tracepoint event] syscalls:sys_exit_unlinkat [Tracepoint event] syscalls:sys_exit_unshare [Tracepoint event] syscalls:sys_exit_ustat [Tracepoint event] syscalls:sys_exit_utime [Tracepoint event] syscalls:sys_exit_utimensat [Tracepoint event] syscalls:sys_exit_utimes [Tracepoint event] syscalls:sys_exit_vhangup [Tracepoint event] syscalls:sys_exit_vmsplice [Tracepoint event] syscalls:sys_exit_wait4 [Tracepoint event] syscalls:sys_exit_waitid [Tracepoint event] syscalls:sys_exit_write [Tracepoint event] syscalls:sys_exit_writev [Tracepoint event] task:task_newtask [Tracepoint event] task:task_rename [Tracepoint event] thermal:cdev_update [Tracepoint event] thermal:thermal_power_cpu_get_power [Tracepoint event] thermal:thermal_power_cpu_limit [Tracepoint event] thermal:thermal_temperature [Tracepoint event] thermal:thermal_zone_trip [Tracepoint event] timer:hrtimer_cancel [Tracepoint event] timer:hrtimer_expire_entry [Tracepoint event] timer:hrtimer_expire_exit [Tracepoint event] timer:hrtimer_init [Tracepoint event] timer:hrtimer_start [Tracepoint event] timer:itimer_expire [Tracepoint event] timer:itimer_state [Tracepoint event] timer:tick_stop [Tracepoint event] timer:timer_cancel [Tracepoint event] timer:timer_expire_entry [Tracepoint event] timer:timer_expire_exit [Tracepoint event] timer:timer_init [Tracepoint event] timer:timer_start [Tracepoint event] tlb:tlb_flush [Tracepoint event] udp:udp_fail_queue_rcv_skb [Tracepoint event] vmscan:mm_shrink_slab_end [Tracepoint event] vmscan:mm_shrink_slab_start [Tracepoint event] vmscan:mm_vmscan_direct_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_direct_reclaim_end [Tracepoint event] vmscan:mm_vmscan_kswapd_sleep [Tracepoint event] vmscan:mm_vmscan_kswapd_wake [Tracepoint event] vmscan:mm_vmscan_lru_isolate [Tracepoint event] vmscan:mm_vmscan_lru_shrink_inactive [Tracepoint event] vmscan:mm_vmscan_memcg_isolate [Tracepoint event] vmscan:mm_vmscan_memcg_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_memcg_reclaim_end [Tracepoint event] vmscan:mm_vmscan_memcg_softlimit_reclaim_begin [Tracepoint event] vmscan:mm_vmscan_memcg_softlimit_reclaim_end [Tracepoint event] vmscan:mm_vmscan_wakeup_kswapd [Tracepoint event] vmscan:mm_vmscan_writepage [Tracepoint event] vsyscall:emulate_vsyscall [Tracepoint event] workqueue:workqueue_activate_work [Tracepoint event] workqueue:workqueue_execute_end [Tracepoint event] workqueue:workqueue_execute_start [Tracepoint event] workqueue:workqueue_queue_work [Tracepoint event] writeback:balance_dirty_pages [Tracepoint event] writeback:bdi_dirty_ratelimit [Tracepoint event] writeback:global_dirty_state [Tracepoint event] writeback:wbc_writepage [Tracepoint event] writeback:writeback_bdi_register [Tracepoint event] writeback:writeback_congestion_wait [Tracepoint event] writeback:writeback_dirty_inode [Tracepoint event] writeback:writeback_dirty_inode_enqueue [Tracepoint event] writeback:writeback_dirty_inode_start [Tracepoint event] writeback:writeback_dirty_page [Tracepoint event] writeback:writeback_exec [Tracepoint event] writeback:writeback_lazytime [Tracepoint event] writeback:writeback_lazytime_iput [Tracepoint event] writeback:writeback_mark_inode_dirty [Tracepoint event] writeback:writeback_nowork [Tracepoint event] writeback:writeback_pages_written [Tracepoint event] writeback:writeback_queue [Tracepoint event] writeback:writeback_queue_io [Tracepoint event] writeback:writeback_sb_inodes_requeue [Tracepoint event] writeback:writeback_single_inode [Tracepoint event] writeback:writeback_single_inode_start [Tracepoint event] writeback:writeback_start [Tracepoint event] writeback:writeback_wait [Tracepoint event] writeback:writeback_wait_iff_congested [Tracepoint event] writeback:writeback_wake_background [Tracepoint event] writeback:writeback_write_inode [Tracepoint event] writeback:writeback_write_inode_start [Tracepoint event] writeback:writeback_written [Tracepoint event] xen:xen_cpu_load_idt [Tracepoint event] xen:xen_cpu_set_ldt [Tracepoint event] xen:xen_cpu_write_gdt_entry [Tracepoint event] xen:xen_cpu_write_idt_entry [Tracepoint event] xen:xen_cpu_write_ldt_entry [Tracepoint event] xen:xen_mc_batch [Tracepoint event] xen:xen_mc_callback [Tracepoint event] xen:xen_mc_entry [Tracepoint event] xen:xen_mc_entry_alloc [Tracepoint event] xen:xen_mc_extend_args [Tracepoint event] xen:xen_mc_flush [Tracepoint event] xen:xen_mc_flush_reason [Tracepoint event] xen:xen_mc_issue [Tracepoint event] xen:xen_mmu_alloc_ptpage [Tracepoint event] xen:xen_mmu_flush_tlb [Tracepoint event] xen:xen_mmu_flush_tlb_all [Tracepoint event] xen:xen_mmu_flush_tlb_others [Tracepoint event] xen:xen_mmu_flush_tlb_single [Tracepoint event] xen:xen_mmu_pgd_clear [Tracepoint event] xen:xen_mmu_pgd_pin [Tracepoint event] xen:xen_mmu_pgd_unpin [Tracepoint event] xen:xen_mmu_pmd_clear [Tracepoint event] xen:xen_mmu_pte_clear [Tracepoint event] xen:xen_mmu_ptep_modify_prot_commit [Tracepoint event] xen:xen_mmu_ptep_modify_prot_start [Tracepoint event] xen:xen_mmu_pud_clear [Tracepoint event] xen:xen_mmu_release_ptpage [Tracepoint event] xen:xen_mmu_set_domain_pte [Tracepoint event] xen:xen_mmu_set_pgd [Tracepoint event] xen:xen_mmu_set_pmd [Tracepoint event] xen:xen_mmu_set_pte [Tracepoint event] xen:xen_mmu_set_pte_at [Tracepoint event] xen:xen_mmu_set_pte_atomic [Tracepoint event] xen:xen_mmu_set_pud [Tracepoint event] xen:xen_mmu_write_cr3 [Tracepoint event] xfs:xfs_agf [Tracepoint event] xfs:xfs_ail_delete [Tracepoint event] xfs:xfs_ail_flushing [Tracepoint event] xfs:xfs_ail_insert [Tracepoint event] xfs:xfs_ail_locked [Tracepoint event] xfs:xfs_ail_move [Tracepoint event] xfs:xfs_ail_pinned [Tracepoint event] xfs:xfs_ail_push [Tracepoint event] xfs:xfs_alloc_exact_done [Tracepoint event] xfs:xfs_alloc_exact_error [Tracepoint event] xfs:xfs_alloc_exact_notfound [Tracepoint event] xfs:xfs_alloc_file_space [Tracepoint event] xfs:xfs_alloc_near_busy [Tracepoint event] xfs:xfs_alloc_near_error [Tracepoint event] xfs:xfs_alloc_near_first [Tracepoint event] xfs:xfs_alloc_near_greater [Tracepoint event] xfs:xfs_alloc_near_lesser [Tracepoint event] xfs:xfs_alloc_near_noentry [Tracepoint event] xfs:xfs_alloc_near_nominleft [Tracepoint event] xfs:xfs_alloc_read_agf [Tracepoint event] xfs:xfs_alloc_size_busy [Tracepoint event] xfs:xfs_alloc_size_done [Tracepoint event] xfs:xfs_alloc_size_error [Tracepoint event] xfs:xfs_alloc_size_neither [Tracepoint event] xfs:xfs_alloc_size_noentry [Tracepoint event] xfs:xfs_alloc_size_nominleft [Tracepoint event] xfs:xfs_alloc_small_done [Tracepoint event] xfs:xfs_alloc_small_error [Tracepoint event] xfs:xfs_alloc_small_freelist [Tracepoint event] xfs:xfs_alloc_small_notenough [Tracepoint event] xfs:xfs_alloc_vextent_allfailed [Tracepoint event] xfs:xfs_alloc_vextent_badargs [Tracepoint event] xfs:xfs_alloc_vextent_loopfailed [Tracepoint event] xfs:xfs_alloc_vextent_noagbp [Tracepoint event] xfs:xfs_alloc_vextent_nofix [Tracepoint event] xfs:xfs_attr_fillstate [Tracepoint event] xfs:xfs_attr_leaf_add [Tracepoint event] xfs:xfs_attr_leaf_add_new [Tracepoint event] xfs:xfs_attr_leaf_add_old [Tracepoint event] xfs:xfs_attr_leaf_add_work [Tracepoint event] xfs:xfs_attr_leaf_addname [Tracepoint event] xfs:xfs_attr_leaf_clearflag [Tracepoint event] xfs:xfs_attr_leaf_compact [Tracepoint event] xfs:xfs_attr_leaf_create [Tracepoint event] xfs:xfs_attr_leaf_flipflags [Tracepoint event] xfs:xfs_attr_leaf_get [Tracepoint event] xfs:xfs_attr_leaf_list [Tracepoint event] xfs:xfs_attr_leaf_lookup [Tracepoint event] xfs:xfs_attr_leaf_rebalance [Tracepoint event] xfs:xfs_attr_leaf_remove [Tracepoint event] xfs:xfs_attr_leaf_removename [Tracepoint event] xfs:xfs_attr_leaf_replace [Tracepoint event] xfs:xfs_attr_leaf_setflag [Tracepoint event] xfs:xfs_attr_leaf_split [Tracepoint event] xfs:xfs_attr_leaf_split_after [Tracepoint event] xfs:xfs_attr_leaf_split_before [Tracepoint event] xfs:xfs_attr_leaf_to_node [Tracepoint event] xfs:xfs_attr_leaf_to_sf [Tracepoint event] xfs:xfs_attr_leaf_toosmall [Tracepoint event] xfs:xfs_attr_leaf_unbalance [Tracepoint event] xfs:xfs_attr_list_add [Tracepoint event] xfs:xfs_attr_list_full [Tracepoint event] xfs:xfs_attr_list_leaf [Tracepoint event] xfs:xfs_attr_list_leaf_end [Tracepoint event] xfs:xfs_attr_list_node_descend [Tracepoint event] xfs:xfs_attr_list_notfound [Tracepoint event] xfs:xfs_attr_list_sf [Tracepoint event] xfs:xfs_attr_list_sf_all [Tracepoint event] xfs:xfs_attr_list_wrong_blk [Tracepoint event] xfs:xfs_attr_node_addname [Tracepoint event] xfs:xfs_attr_node_get [Tracepoint event] xfs:xfs_attr_node_list [Tracepoint event] xfs:xfs_attr_node_lookup [Tracepoint event] xfs:xfs_attr_node_removename [Tracepoint event] xfs:xfs_attr_node_replace [Tracepoint event] xfs:xfs_attr_refillstate [Tracepoint event] xfs:xfs_attr_rmtval_get [Tracepoint event] xfs:xfs_attr_rmtval_remove [Tracepoint event] xfs:xfs_attr_rmtval_set [Tracepoint event] xfs:xfs_attr_sf_add [Tracepoint event] xfs:xfs_attr_sf_addname [Tracepoint event] xfs:xfs_attr_sf_create [Tracepoint event] xfs:xfs_attr_sf_lookup [Tracepoint event] xfs:xfs_attr_sf_remove [Tracepoint event] xfs:xfs_attr_sf_removename [Tracepoint event] xfs:xfs_attr_sf_to_leaf [Tracepoint event] xfs:xfs_bdstrat_shut [Tracepoint event] xfs:xfs_bmap_post_update [Tracepoint event] xfs:xfs_bmap_pre_update [Tracepoint event] xfs:xfs_btree_corrupt [Tracepoint event] xfs:xfs_buf_bawrite [Tracepoint event] xfs:xfs_buf_delwri_queue [Tracepoint event] xfs:xfs_buf_delwri_queued [Tracepoint event] xfs:xfs_buf_delwri_split [Tracepoint event] xfs:xfs_buf_error_relse [Tracepoint event] xfs:xfs_buf_find [Tracepoint event] xfs:xfs_buf_free [Tracepoint event] xfs:xfs_buf_get [Tracepoint event] xfs:xfs_buf_get_uncached [Tracepoint event] xfs:xfs_buf_hold [Tracepoint event] xfs:xfs_buf_init [Tracepoint event] xfs:xfs_buf_iodone [Tracepoint event] xfs:xfs_buf_ioerror [Tracepoint event] xfs:xfs_buf_iowait [Tracepoint event] xfs:xfs_buf_iowait_done [Tracepoint event] xfs:xfs_buf_item_committed [Tracepoint event] xfs:xfs_buf_item_format [Tracepoint event] xfs:xfs_buf_item_format_ordered [Tracepoint event] xfs:xfs_buf_item_format_stale [Tracepoint event] xfs:xfs_buf_item_iodone [Tracepoint event] xfs:xfs_buf_item_iodone_async [Tracepoint event] xfs:xfs_buf_item_ordered [Tracepoint event] xfs:xfs_buf_item_pin [Tracepoint event] xfs:xfs_buf_item_push [Tracepoint event] xfs:xfs_buf_item_relse [Tracepoint event] xfs:xfs_buf_item_size [Tracepoint event] xfs:xfs_buf_item_size_ordered [Tracepoint event] xfs:xfs_buf_item_size_stale [Tracepoint event] xfs:xfs_buf_item_unlock [Tracepoint event] xfs:xfs_buf_item_unlock_stale [Tracepoint event] xfs:xfs_buf_item_unpin [Tracepoint event] xfs:xfs_buf_item_unpin_stale [Tracepoint event] xfs:xfs_buf_lock [Tracepoint event] xfs:xfs_buf_lock_done [Tracepoint event] xfs:xfs_buf_read [Tracepoint event] xfs:xfs_buf_rele [Tracepoint event] xfs:xfs_buf_submit [Tracepoint event] xfs:xfs_buf_submit_wait [Tracepoint event] xfs:xfs_buf_trylock [Tracepoint event] xfs:xfs_buf_unlock [Tracepoint event] xfs:xfs_buf_wait_buftarg [Tracepoint event] xfs:xfs_bunmap [Tracepoint event] xfs:xfs_collapse_file_space [Tracepoint event] xfs:xfs_create [Tracepoint event] xfs:xfs_da_btree_corrupt [Tracepoint event] xfs:xfs_da_fixhashpath [Tracepoint event] xfs:xfs_da_grow_inode [Tracepoint event] xfs:xfs_da_join [Tracepoint event] xfs:xfs_da_link_after [Tracepoint event] xfs:xfs_da_link_before [Tracepoint event] xfs:xfs_da_node_add [Tracepoint event] xfs:xfs_da_node_create [Tracepoint event] xfs:xfs_da_node_rebalance [Tracepoint event] xfs:xfs_da_node_remove [Tracepoint event] xfs:xfs_da_node_split [Tracepoint event] xfs:xfs_da_node_toosmall [Tracepoint event] xfs:xfs_da_node_unbalance [Tracepoint event] xfs:xfs_da_path_shift [Tracepoint event] xfs:xfs_da_root_join [Tracepoint event] xfs:xfs_da_root_split [Tracepoint event] xfs:xfs_da_shrink_inode [Tracepoint event] xfs:xfs_da_split [Tracepoint event] xfs:xfs_da_swap_lastblock [Tracepoint event] xfs:xfs_da_unlink_back [Tracepoint event] xfs:xfs_da_unlink_forward [Tracepoint event] xfs:xfs_delalloc_enospc [Tracepoint event] xfs:xfs_destroy_inode [Tracepoint event] xfs:xfs_dir2_block_addname [Tracepoint event] xfs:xfs_dir2_block_lookup [Tracepoint event] xfs:xfs_dir2_block_removename [Tracepoint event] xfs:xfs_dir2_block_replace [Tracepoint event] xfs:xfs_dir2_block_to_leaf [Tracepoint event] xfs:xfs_dir2_block_to_sf [Tracepoint event] xfs:xfs_dir2_grow_inode [Tracepoint event] xfs:xfs_dir2_leaf_addname [Tracepoint event] xfs:xfs_dir2_leaf_lookup [Tracepoint event] xfs:xfs_dir2_leaf_removename [Tracepoint event] xfs:xfs_dir2_leaf_replace [Tracepoint event] xfs:xfs_dir2_leaf_to_block [Tracepoint event] xfs:xfs_dir2_leaf_to_node [Tracepoint event] xfs:xfs_dir2_leafn_add [Tracepoint event] xfs:xfs_dir2_leafn_moveents [Tracepoint event] xfs:xfs_dir2_leafn_remove [Tracepoint event] xfs:xfs_dir2_node_addname [Tracepoint event] xfs:xfs_dir2_node_lookup [Tracepoint event] xfs:xfs_dir2_node_removename [Tracepoint event] xfs:xfs_dir2_node_replace [Tracepoint event] xfs:xfs_dir2_node_to_leaf [Tracepoint event] xfs:xfs_dir2_sf_addname [Tracepoint event] xfs:xfs_dir2_sf_create [Tracepoint event] xfs:xfs_dir2_sf_lookup [Tracepoint event] xfs:xfs_dir2_sf_removename [Tracepoint event] xfs:xfs_dir2_sf_replace [Tracepoint event] xfs:xfs_dir2_sf_to_block [Tracepoint event] xfs:xfs_dir2_sf_toino4 [Tracepoint event] xfs:xfs_dir2_sf_toino8 [Tracepoint event] xfs:xfs_dir2_shrink_inode [Tracepoint event] xfs:xfs_dir_fsync [Tracepoint event] xfs:xfs_discard_busy [Tracepoint event] xfs:xfs_discard_exclude [Tracepoint event] xfs:xfs_discard_extent [Tracepoint event] xfs:xfs_discard_toosmall [Tracepoint event] xfs:xfs_dqadjust [Tracepoint event] xfs:xfs_dqalloc [Tracepoint event] xfs:xfs_dqattach_found [Tracepoint event] xfs:xfs_dqattach_get [Tracepoint event] xfs:xfs_dqflush [Tracepoint event] xfs:xfs_dqflush_done [Tracepoint event] xfs:xfs_dqflush_force [Tracepoint event] xfs:xfs_dqget_dup [Tracepoint event] xfs:xfs_dqget_freeing [Tracepoint event] xfs:xfs_dqget_hit [Tracepoint event] xfs:xfs_dqget_miss [Tracepoint event] xfs:xfs_dqput [Tracepoint event] xfs:xfs_dqput_free [Tracepoint event] xfs:xfs_dqput_wait [Tracepoint event] xfs:xfs_dqread [Tracepoint event] xfs:xfs_dqread_fail [Tracepoint event] xfs:xfs_dqreclaim_busy [Tracepoint event] xfs:xfs_dqreclaim_dirty [Tracepoint event] xfs:xfs_dqreclaim_done [Tracepoint event] xfs:xfs_dqreclaim_want [Tracepoint event] xfs:xfs_dqrele [Tracepoint event] xfs:xfs_dqtobp_read [Tracepoint event] xfs:xfs_dquot_dqalloc [Tracepoint event] xfs:xfs_dquot_dqdetach [Tracepoint event] xfs:xfs_evict_inode [Tracepoint event] xfs:xfs_extent_busy [Tracepoint event] xfs:xfs_extent_busy_clear [Tracepoint event] xfs:xfs_extent_busy_enomem [Tracepoint event] xfs:xfs_extent_busy_force [Tracepoint event] xfs:xfs_extent_busy_reuse [Tracepoint event] xfs:xfs_extent_busy_trim [Tracepoint event] xfs:xfs_extlist [Tracepoint event] xfs:xfs_file_buffered_write [Tracepoint event] xfs:xfs_file_compat_ioctl [Tracepoint event] xfs:xfs_file_direct_write [Tracepoint event] xfs:xfs_file_fsync [Tracepoint event] xfs:xfs_file_ioctl [Tracepoint event] xfs:xfs_file_read [Tracepoint event] xfs:xfs_file_splice_read [Tracepoint event] xfs:xfs_filemap_fault [Tracepoint event] xfs:xfs_filemap_page_mkwrite [Tracepoint event] xfs:xfs_filestream_free [Tracepoint event] xfs:xfs_filestream_lookup [Tracepoint event] xfs:xfs_filestream_pick [Tracepoint event] xfs:xfs_filestream_scan [Tracepoint event] xfs:xfs_free_extent [Tracepoint event] xfs:xfs_free_file_space [Tracepoint event] xfs:xfs_gbmap_direct [Tracepoint event] xfs:xfs_gbmap_direct_endio [Tracepoint event] xfs:xfs_gbmap_direct_new [Tracepoint event] xfs:xfs_gbmap_direct_none [Tracepoint event] xfs:xfs_gbmap_direct_update [Tracepoint event] xfs:xfs_get_acl [Tracepoint event] xfs:xfs_get_blocks_alloc [Tracepoint event] xfs:xfs_get_blocks_found [Tracepoint event] xfs:xfs_get_blocks_notfound [Tracepoint event] xfs:xfs_getattr [Tracepoint event] xfs:xfs_ialloc_read_agi [Tracepoint event] xfs:xfs_iext_insert [Tracepoint event] xfs:xfs_iext_remove [Tracepoint event] xfs:xfs_iget_hit [Tracepoint event] xfs:xfs_iget_miss [Tracepoint event] xfs:xfs_iget_reclaim [Tracepoint event] xfs:xfs_iget_reclaim_fail [Tracepoint event] xfs:xfs_iget_skip [Tracepoint event] xfs:xfs_ihold [Tracepoint event] xfs:xfs_ilock [Tracepoint event] xfs:xfs_ilock_demote [Tracepoint event] xfs:xfs_ilock_nowait [Tracepoint event] xfs:xfs_inactive_symlink [Tracepoint event] xfs:xfs_inode_clear_eofblocks_tag [Tracepoint event] xfs:xfs_inode_free_eofblocks_invalid [Tracepoint event] xfs:xfs_inode_item_push [Tracepoint event] xfs:xfs_inode_pin [Tracepoint event] xfs:xfs_inode_set_eofblocks_tag [Tracepoint event] xfs:xfs_inode_unpin [Tracepoint event] xfs:xfs_inode_unpin_nowait [Tracepoint event] xfs:xfs_insert_file_space [Tracepoint event] xfs:xfs_invalidatepage [Tracepoint event] xfs:xfs_ioctl_setattr [Tracepoint event] xfs:xfs_iomap_prealloc_size [Tracepoint event] xfs:xfs_irec_merge_post [Tracepoint event] xfs:xfs_irec_merge_pre [Tracepoint event] xfs:xfs_irele [Tracepoint event] xfs:xfs_itruncate_extents_end [Tracepoint event] xfs:xfs_itruncate_extents_start [Tracepoint event] xfs:xfs_iunlock [Tracepoint event] xfs:xfs_link [Tracepoint event] xfs:xfs_log_assign_tail_lsn [Tracepoint event] xfs:xfs_log_done_nonperm [Tracepoint event] xfs:xfs_log_done_perm [Tracepoint event] xfs:xfs_log_force [Tracepoint event] xfs:xfs_log_grant_sleep [Tracepoint event] xfs:xfs_log_grant_wake [Tracepoint event] xfs:xfs_log_grant_wake_up [Tracepoint event] xfs:xfs_log_recover_buf_cancel [Tracepoint event] xfs:xfs_log_recover_buf_cancel_add [Tracepoint event] xfs:xfs_log_recover_buf_cancel_ref_inc [Tracepoint event] xfs:xfs_log_recover_buf_dquot_buf [Tracepoint event] xfs:xfs_log_recover_buf_inode_buf [Tracepoint event] xfs:xfs_log_recover_buf_not_cancel [Tracepoint event] xfs:xfs_log_recover_buf_recover [Tracepoint event] xfs:xfs_log_recover_buf_reg_buf [Tracepoint event] xfs:xfs_log_recover_inode_cancel [Tracepoint event] xfs:xfs_log_recover_inode_recover [Tracepoint event] xfs:xfs_log_recover_inode_skip [Tracepoint event] xfs:xfs_log_recover_item_add [Tracepoint event] xfs:xfs_log_recover_item_add_cont [Tracepoint event] xfs:xfs_log_recover_item_recover [Tracepoint event] xfs:xfs_log_recover_item_reorder_head [Tracepoint event] xfs:xfs_log_recover_item_reorder_tail [Tracepoint event] xfs:xfs_log_regrant [Tracepoint event] xfs:xfs_log_regrant_exit [Tracepoint event] xfs:xfs_log_regrant_reserve_enter [Tracepoint event] xfs:xfs_log_regrant_reserve_exit [Tracepoint event] xfs:xfs_log_regrant_reserve_sub [Tracepoint event] xfs:xfs_log_reserve [Tracepoint event] xfs:xfs_log_reserve_exit [Tracepoint event] xfs:xfs_log_umount_write [Tracepoint event] xfs:xfs_log_ungrant_enter [Tracepoint event] xfs:xfs_log_ungrant_exit [Tracepoint event] xfs:xfs_log_ungrant_sub [Tracepoint event] xfs:xfs_lookup [Tracepoint event] xfs:xfs_map_blocks_alloc [Tracepoint event] xfs:xfs_map_blocks_found [Tracepoint event] xfs:xfs_pagecache_inval [Tracepoint event] xfs:xfs_perag_clear_eofblocks [Tracepoint event] xfs:xfs_perag_clear_reclaim [Tracepoint event] xfs:xfs_perag_get [Tracepoint event] xfs:xfs_perag_get_tag [Tracepoint event] xfs:xfs_perag_put [Tracepoint event] xfs:xfs_perag_set_eofblocks [Tracepoint event] xfs:xfs_perag_set_reclaim [Tracepoint event] xfs:xfs_read_agf [Tracepoint event] xfs:xfs_read_agi [Tracepoint event] xfs:xfs_readdir [Tracepoint event] xfs:xfs_readlink [Tracepoint event] xfs:xfs_releasepage [Tracepoint event] xfs:xfs_remove [Tracepoint event] xfs:xfs_rename [Tracepoint event] xfs:xfs_reset_dqcounts [Tracepoint event] xfs:xfs_setattr [Tracepoint event] xfs:xfs_setfilesize [Tracepoint event] xfs:xfs_swap_extent_after [Tracepoint event] xfs:xfs_swap_extent_before [Tracepoint event] xfs:xfs_symlink [Tracepoint event] xfs:xfs_trans_bhold [Tracepoint event] xfs:xfs_trans_bhold_release [Tracepoint event] xfs:xfs_trans_binval [Tracepoint event] xfs:xfs_trans_bjoin [Tracepoint event] xfs:xfs_trans_brelse [Tracepoint event] xfs:xfs_trans_buf_ordered [Tracepoint event] xfs:xfs_trans_commit_lsn [Tracepoint event] xfs:xfs_trans_get_buf [Tracepoint event] xfs:xfs_trans_get_buf_recur [Tracepoint event] xfs:xfs_trans_getsb [Tracepoint event] xfs:xfs_trans_getsb_recur [Tracepoint event] xfs:xfs_trans_log_buf [Tracepoint event] xfs:xfs_trans_read_buf [Tracepoint event] xfs:xfs_trans_read_buf_io [Tracepoint event] xfs:xfs_trans_read_buf_recur [Tracepoint event] xfs:xfs_trans_read_buf_shut [Tracepoint event] xfs:xfs_unwritten_convert [Tracepoint event] xfs:xfs_update_time [Tracepoint event] xfs:xfs_vm_bmap [Tracepoint event] xfs:xfs_writepage [Tracepoint event] xfs:xfs_zero_file_space [Tracepoint event] xhci-hcd:xhci_address_ctx [Tracepoint event] xhci-hcd:xhci_cmd_completion [Tracepoint event] xhci-hcd:xhci_dbg_address [Tracepoint event] xhci-hcd:xhci_dbg_cancel_urb [Tracepoint event] xhci-hcd:xhci_dbg_context_change [Tracepoint event] xhci-hcd:xhci_dbg_init [Tracepoint event] xhci-hcd:xhci_dbg_quirks [Tracepoint event] xhci-hcd:xhci_dbg_reset_ep [Tracepoint event] xhci-hcd:xhci_dbg_ring_expansion [Tracepoint event]

pipcet commented 8 years ago

Those tick numbers are way too high. They do increase as they should, though, so we're counting something...

Can you try changing LWP_FLAGS (the first instance thereof) to 0x80000008L in rr/src/LWP.h, to match the value in the kernel file? If that doesn't help, it might be time to uncomment all the //printf's in PerfCounters.cc and Task.cc to see what's happening.

edwintorok commented 8 years ago

On 2016-07-10 16:25, pipcet wrote:

Those tick numbers are way too high. They do increase as they should, though, so we're counting something...

Can you try changing LWP_FLAGS (the first instance thereof) to 0x80000008L in rr/src/LWP.h, to match the value in the kernel file?

New dump, still looks too high: https://gist.githubusercontent.com/anonymous/a05a08a276ac815074dabc02d0dc6765/raw/91ac4ea2397ed899cdb3333e61b0c6c9f2107ff6/dump2

If that doesn't help, it might be time to uncomment all the //printf's in PerfCounters.cc and Task.cc to see what's happening.

I enabled some printfs, see here for output: https://gist.githubusercontent.com/anonymous/a05a08a276ac815074dabc02d0dc6765/raw/91ac4ea2397ed899cdb3333e61b0c6c9f2107ff6/record2

pipcet commented 8 years ago

Hmm. It's suspicious that the XSAVE mask word is always 0x4000000000000007. I'll have another look tomorrow, but I've pushed one last set of changes that just might fix things, if we're lucky.

edwintorok commented 8 years ago

On 2016-07-10 19:01, pipcet wrote:

Hmm. It's suspicious that the XSAVE mask word is always 0x4000000000000007. I'll have another look tomorrow, but I've pushed one last set of changes that just might fix things, if we're lucky.

New output, it still complains about mismatching ticks:

rr record: https://gist.githubusercontent.com/anonymous/189f8c7c17c8ed04309fe38dd16dd786/raw/390d135140f9fce4ee5e5bb89f357b5f81bd45fd/gistfile1.txt

basic_test.run: https://gist.githubusercontent.com/anonymous/7c8ce3cb1af7a9b73435ff031af46000/raw/7012299ba278c5dc7e4378bc1715e4a88d868b30/basic_test.run

rr dump: https://gist.githubusercontent.com/anonymous/c893f96270704e0d1b88379a97757d0c/raw/6a4e9d87e64d76055712dfafda3f17ab80b66074/gistfile1.txt

pipcet commented 8 years ago

Thanks again for all the testing!

Okay, that's strange. We're attempting to reload the LWP area from the XSAVE area, but the flags end up either being 0 (which resets the filter word to 0x38000000, for some reason) or 8 (no interrupts), rather than 0x80000008. However, we do see interrupts even though the XSAVE area claims they're disabled.

wrote ER: 7 40000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 70001000 0 80000008 0 70002000 0 2000 28000000 0 0 0 0 0 0 0 0 0 0 2702a 0 0 0 0 0 0 0 0 0 0 0 0 0
read ER: 7 40000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 70001000 0 8 0 70002000 0 2000 28000000 0 0 0 0 0 0 0 0 0 0 26d94 0 0 0 0 0 0 0 0 0 0 0 0 0

That's definitely wrong, but it's wrong in two different ways, so it must depend on state we're not dumping to the debug log. We can deal with the interrupt flag in the XSAVE area not being what it should be, as long as the actual interrupts come in; we can't deal with LWP disabling itself randomly, so it all hinges on finding out whether it's truly random or predictable somehow. It's not undoable to put a loop around the XRSTOR that's failing to restore state for some reason, if it's truly random...

I'll try putting in some more tests.

pipcet commented 8 years ago

so it must depend on state we're not dumping to the debug log

I'm really quite confident that must be the LWPCB, then, which my CPU seems to ignore except for the counter reset value, but yours might not. I've pushed code to dump that, which produces a lot of output, but it'd be great to have a look at that if and when you find the time to test.

Thanks again for all the testing, sorry it's taking longer than expected to get this working on your CPU!

edwintorok commented 8 years ago

On 2016-07-12 22:48, pipcet wrote:

so it must depend on state we're not dumping to the debug log

I'm really quite confident that must be the LWPCB, then, which my CPU seems to ignore except for the counter reset value, but yours might not. I've pushed code to dump that, which produces a lot of output, but it'd be great to have a look at that if and when you find the time to test.

No problem, as long as I don't have to patch the kernel I can test more easily, here you go: https://gist.githubusercontent.com/anonymous/2533d56282601a1f09a83040a61a84db/raw/4391b415042a9b5e428e1a1f727cf65665757e47/gistfile1.txt

Thanks again for all the testing, sorry it's taking longer than expected to get this working on your CPU!

I'm not in a particular hurry, thanks for the patches :)

pipcet commented 8 years ago

No problem, as long as I don't have to patch the kernel I can test more easily

Excellent. Further kernel patching should not be required to get things working, though the ultimate official kernel driver will have to work slightly differently.

Okay, it's not the LWPCB; I've pushed another version to print the user-defined bytes of the XSAVE context, can you produce another log?

Thanks!

edwintorok commented 8 years ago

On 2016-07-13 01:13, pipcet wrote:

No problem, as long as I don't have to patch the kernel I can test more easily

Excellent. Further kernel patching should not be required to get things working, though the ultimate official kernel driver will have to work slightly differently.

Okay, it's not the LWPCB; I've pushed another version to print the user-defined bytes of the XSAVE context, can you produce another log?

Does this have what you were looking for? https://gist.githubusercontent.com/anonymous/b019d28000eee52147d55bd4a965f8bb/raw/4021367e04703a2d0e05272562b0d14a891b070b/gistfile1.txt

pipcet commented 8 years ago

Thanks again! I'm afraid that's another theory falsified, because the XSAVE context does not differ between calls to PTRACE_SETREGS that fail and those that succeed.

But it's not random either: there's a regular pattern during the main part of the alarm program where we get one interrupt, try to reset the counter, and instead disable it entirely, then re-enable it when we next look at the tracee (the next syscall, that is).

So that means there are two more options: it could depend on the regular, non-extended registers somehow, or there's a bit of state hidden in the CPU that we can't access.

In particular, it might depend on the instruction we're resuming execution at: after a syscall, that's a ret, but, depending on how your code was compiled, after a threshold interrupt, it might be another branch. Just maybe your CPU's microcode confuses "don't count this branch" and "don't start counting branches at all", which would be unfortunate but fixable.

So I'm pushing a patch to dump the standard registers, but I'm also disabling the filter bits in case it's just the filtering mechanism that's going wrong; counting all branches, not just conditional branches, means we lose some compatibility with the Intel version but should otherwise work just as well.

I'm afraid I'm going to have to ask you for another log, but also for the output of objdump -d bin/alarm; that would disassemble the alarm binary and allow me to make sense of the program counter values.

edwintorok commented 8 years ago

On 2016-07-13 18:03, pipcet wrote:

Thanks again! I'm afraid that's another theory falsified, because the XSAVE context does not differ between calls to PTRACE_SETREGS that fail and those that succeed.

But it's not random either: there's a regular pattern during the main part of the alarm program where we get one interrupt, try to reset the counter, and instead disable it entirely, then re-enable it when we next look at the tracee (the next syscall, that is).

So that means there are two more options: it could depend on the regular, non-extended registers somehow, or there's a bit of state hidden in the CPU that we can't access.

In particular, it might depend on the instruction we're resuming execution at: after a syscall, that's a |ret|, but, depending on how your code was compiled, after a threshold interrupt, it might be another branch. Just maybe your CPU's microcode confuses "don't count this branch" and "don't start counting branches at all", which would be unfortunate but fixable.

So I'm pushing a patch to dump the standard registers, but I'm also disabling the filter bits in case it's just the filtering mechanism that's going wrong; counting all branches, not just conditional branches, means we lose some compatibility with the Intel version but should otherwise work just as well.

new rr record output: https://gist.githubusercontent.com/anonymous/d065bda84d6e3b5ee1ee3c820ffde9c4/raw/39a5c54b6aac3327f4c3a7d81559537f324f7f07/gistfile1.txt

I'm afraid I'm going to have to ask you for another log, but also for the output of |objdump -d bin/alarm|; that would disassemble the alarm binary and allow me to make sense of the program counter values.

objdump -d ./alarm: https://gist.githubusercontent.com/anonymous/11df7689c2ab78184a9fd2ead2194fdc/raw/7a5701635cc44a1ba8aeb21f0bcbd57dbd974517/gistfile1.txt

objdump -dS ./alarm https://gist.githubusercontent.com/anonymous/40b4d5eb866eb775d52f814d9e214c05/raw/af1bce804502c469e1106f503c83acc7733bc4eb/gistfile1.txt

Ah I used clang and ninja to build rr, does that matter?

$ clang version 3.8.0 (tags/RELEASE_380/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

$ uname -a Linux bolt 4.6.3-gentoo-rr #3 SMP Sun Jul 10 11:33:14 EEST 2016 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

My libc is: GNU C Library (Gentoo 2.23-r2 p4) stable release version 2.23, by Roland McGrath et al.

And looks like it was compiled with (according to -frecord-gcc-switches): GNU C11 5.4.0 -march=bdver2 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -msse4a -mcx16 -msahf -mno-movbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mlwp -mfma -mfma4 -mxop -mbmi -mno-bmi2 -mavx -mno-avx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mno-rdrnd -mf16c -mno-fsgsbase -mno-rdseed -mprfchw -mno-adx -mfxsr -mxsave -mno-xsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=16 --param l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=bdver2 -m64 -mno-tbm -ggdb -O2 -std=gnu11 -fgnu89-inline -fmerge-all-constants -fno-stack-protector -fno-strict-aliasing -frounding-math -fPIC -fno-omit-frame-pointer -ftls-model=initial-exec

Thanks,

pipcet commented 8 years ago

Ah I used clang and ninja to build rr, does that matter?

I think that's fine. Thanks for providing the versions, I would have asked for those soon anyway :-)

Okay, no luck with disabling the filtering. I just pushed a horrible hack to force most resumptions to start at the ret instruction on the rr page rather than in actual user code. It's possible that produces different output, though it's also possible you'll get a segfault because I messed up somehow.

Can you provide the rr record log again?

Thanks!

edwintorok commented 8 years ago

On 2016-07-13 19:34, pipcet wrote:

Ah I used clang and ninja to build rr, does that matter?

I think that's fine. Thanks for providing the versions, I would have asked for those soon anyway :-)

Okay, no luck with disabling the filtering. I just pushed a horrible hack to force most resumptions to start at the |ret| instruction on the rr page rather than in actual user code. It's possible that produces different output, though it's also possible you'll get a segfault because I messed up somehow.

Can you provide the rr record log again?

rr record (no segfault): https://gist.githubusercontent.com/anonymous/3465881784d8aef630282bbeed88afa9/raw/e9e20dd18d5a194dcdd3f3f7de43768367b0c4d4/gistfile1.txt

objdump is same as before.

pipcet commented 8 years ago

Argh, sorry, I did mess up, just in a different way (the change was a nop because of a typo :-(). Fixed now, if you want to try again.

edwintorok commented 8 years ago

On 2016-07-13 22:23, pipcet wrote:

Argh, sorry, I did mess up, just in a different way (the change was a nop because of a typo :-(). Fixed now, if you want to try again.

Better?

https://gist.githubusercontent.com/anonymous/329c5b1c18f3ab38d103c6bea1aac55c/raw/044d6515e90b8bd642e4353ca87d1b1ac1982592/gistfile1.txt

pipcet commented 8 years ago

Okay, we're getting somewhere. One of two conditions in which the LWP status gets cleared is when the resume flag is set in eflags. The other appears to be depending on the differences when we resume from syscalls versus signals, but not the RIP. I think both of those should be fixable, but I need to clarify the second condition first.

Can you recompile rr with the new changes and provide the output of:

RR_LOG=all:debug strace -e ptrace,wait4 ./bin/rr record ./bin/alarm > log.txt 2>&1

? It might be too large, but it should compress very well if that's the problem. If we're lucky, this is the last diagnostic request, thanks for all your patience.

edwintorok commented 8 years ago

On 2016-07-14 09:03, pipcet wrote:

|RR_LOG=all:debug strace -e ptrace,wait4 ./bin/rr record ./bin/alarm > log.txt 2>&1|

https://gist.github.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/0413c4fc74eb47ae268d460a1becfb3575bc3960/log.txt.xz

pipcet commented 8 years ago

Thank you! It still seems like we're restoring identical-looking CPU context and getting different results depending on whether our process stopped because of a signal or because of a syscall. I'm still trying to understand that; the really strange thing is how it depends on the CPU. I've modified alarm.c to detect when the LWP state is disabled, just in case it happens after the PTRACE_CONT. That would be nice to exclude, though I'd be really surprised if it turned out something else is disabling LWP while the process is running.

Can you provide another log like the last one, with rr and the alarm binary rebuilt from the current branch?

edwintorok commented 8 years ago

On 2016-07-14 18:20, pipcet wrote:

Thank you! It still seems like we're restoring identical-looking CPU context and getting different results depending on whether our process stopped because of a signal or because of a syscall. I'm still trying to understand that; the really strange thing is how it depends on the CPU. I've modified alarm.c to detect when the LWP state is disabled, just in case it happens after the PTRACE_CONT. That would be nice to exclude, though I'd be really surprised if it turned out something else is disabling LWP while the process is running.

Can you provide another log like the last one, with rr and the alarm binary rebuilt from the current branch?

/usr/bin/clang -DRR_VERSION=\"4.3.0\" -I/home/edwin/rr/rr/include -I. -pthread -g3 -Wall -Wextra -Wstrict-prototypes -Wno-unused-command-line-argument -O0 -Werror -O0 -Werror -MD -MT CMakeFiles/alarm.dir/src/test/alarm.c.o -MF CMakeFiles/alarm.dir/src/test/alarm.c.o.d -o CMakeFiles/alarm.dir/src/test/alarm.c.o -c /home/edwin/rr/rr/src/test/alarm.c /home/edwin/rr/rr/src/test/alarm.c:30:18: error: invalid instruction mnemonic 'slwpcb' asm volatile("slwpcb %0" : "=r" (out) : : "flags", "memory"); ^

:1:2: note: instantiated into assembly here slwpcb %rax ^ /home/edwin/rr/rr/src/test/alarm.c:36:20: error: invalid instruction mnemonic 'llwpcb' asm volatile("llwpcb %0" : : "r" (0x70001000) : "flags", "memory"); ^ :1:2: note: instantiated into assembly here llwpcb %edx ^
edwintorok commented 8 years ago

On 2016-07-14 19:13, Török Edwin wrote:

On 2016-07-14 18:20, pipcet wrote:

Thank you! It still seems like we're restoring identical-looking CPU context and getting different results depending on whether our process stopped because of a signal or because of a syscall. I'm still trying to understand that; the really strange thing is how it depends on the CPU. I've modified alarm.c to detect when the LWP state is disabled, just in case it happens after the PTRACE_CONT. That would be nice to exclude, though I'd be really surprised if it turned out something else is disabling LWP while the process is running.

Can you provide another log like the last one, with rr and the alarm binary rebuilt from the current branch?

/usr/bin/clang -DRR_VERSION=\"4.3.0\" -I/home/edwin/rr/rr/include -I. -pthread -g3 -Wall -Wextra -Wstrict-prototypes -Wno-unused-command-line-argument -O0 -Werror -O0 -Werror -MD -MT CMakeFiles/alarm.dir/src/test/alarm.c.o -MF CMakeFiles/alarm.dir/src/test/alarm.c.o.d -o CMakeFiles/alarm.dir/src/test/alarm.c.o -c /home/edwin/rr/rr/src/test/alarm.c /home/edwin/rr/rr/src/test/alarm.c:30:18: error: invalid instruction mnemonic 'slwpcb' asm volatile("slwpcb %0" : "=r" (out) : : "flags", "memory"); ^

:1:2: note: instantiated into assembly here slwpcb %rax ^ /home/edwin/rr/rr/src/test/alarm.c:36:20: error: invalid instruction mnemonic 'llwpcb' asm volatile("llwpcb %0" : : "r" (0x70001000) : "flags", "memory"); ^ :1:2: note: instantiated into assembly here llwpcb %edx ^

Compiles with GCC.

$ RR_LOG=all:debug strace -e ptrace,wait4 ./bin/rr record ./bin/alarm > log.txt 2>&1 https://gist.github.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/d015b56821db94e2a968189266aa94b7bf067c13/log.txt.xz

$ objdump -dS bin/alarm: https://gist.githubusercontent.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/d015b56821db94e2a968189266aa94b7bf067c13/objdumpds

$ gcc --version gcc (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Keno commented 8 years ago

Seems like the LLVM builtin assembler doesn't understand those instructions. Should be fairly straightforward to add.

pipcet commented 8 years ago

Thanks! Yes, that was it, I changed the instructions to use the .byte form and confirmed they compile with clang now.

Unfortunately, I got confused: LWP doesn't get disabled entirely, it's only the flags that get cleared randomly. Can you try again?

edwintorok commented 8 years ago

On 2016-07-14 19:40, pipcet wrote:

Thanks! Yes, that was it, I changed the instructions to use the .byte form and confirmed they compile with clang now.

Unfortunately, I got confused: LWP doesn't get disabled entirely, it's only the flags that get cleared randomly. Can you try again?

https://gist.github.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/91d6a20b5640c1519ad9a0b121302945737b804d/log.txt.xz

pipcet commented 8 years ago

Okay, so I've decided that it's not really that important (though it is a vexing question) just how things are going wrong, as long as we can fix them somehow. I've just pushed a change that duplicates the relevant information between the LWPCB and the XSAVE area, then calls LLWPCB on returning, in the context of the tracee. That should work on all CPUs that support LWP. Can you produce another log to see if I'm right (two, ideally: one for alarm and one for the version I've named alarm2)?

If that works, I can just rewrite the rest of the code to use the LWPCB and ignore the XSAVE area, which would probably simplify things overall.

This version most likely will most likely still run into the mismatched ticks error, but the rr dump and rr replay -a output might be interesting, too.

Thanks!

edwintorok commented 8 years ago

On 2016-07-14 22:20, pipcet wrote:

Okay, so I've decided that it's not really that important (though it is a vexing question) just how things are going wrong, as long as we can fix them somehow. I've just pushed a change that duplicates the relevant information between the LWPCB and the XSAVE area, then calls LLWPCB on returning, in the context of the tracee. That should work on all CPUs that support LWP. Can you produce another log to see if I'm right (two, ideally: one for |alarm| and one for the version I've named |alarm2|)?

If that works, I can just rewrite the rest of the code to use the LWPCB and ignore the XSAVE area, which would probably simplify things overall.

This version most likely will most likely still run into the mismatched ticks error, but the |rr dump| and |rr replay -a| output might be interesting, too.

I've pushed the logs here: https://gist.github.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73 git clone https://gist.github.com/1b4d95b4cb0b79bd2d85e6f3dc457e73.git

Many thanks for all the debugging!

pipcet commented 8 years ago

Okay, this is taking me longer than I thought it would, but I'm experimenting with using LLWPCB to modify the LWP state; it's still disabled and read using XSAVE, but maybe that's good enough for your CPU.

There's a snapshot at https://github.com/pipcet/rr/tree/lwp-new-20160715 , if you want to try it. It's entirely possible it won't work, but I'd appreciate logs in either case.

edwintorok commented 8 years ago

On 2016-07-16 00:22, pipcet wrote:

Okay, this is taking me longer than I thought it would, but I'm experimenting with using LLWPCB to modify the LWP state; it's still disabled and read using XSAVE, but maybe that's good enough for your CPU.

There's a snapshot at https://github.com/pipcet/rr/tree/lwp-new-20160715 , if you want to try it. It's entirely possible it won't work, but I'd appreciate logs in either case.

Thanks, I've pushed the logs (and the script used to create them) to: https://gist.github.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73

rr record alarm fails with this message now: rr: /home/edwin/rr/rr/src/LWP.cc:272: rr::Ticks rr::LWP::read_ticks_nondestructively(): Assertion `counter_value >= -2048' failed.

Note that alarm2 doesn't finish (uses 100% CPU), I had to kill it after 5s.

pipcet commented 8 years ago

Thank you!

Note that alarm2 doesn't finish (uses 100% CPU), I had to kill it after 5s.

Same here, I'm afraid, I'm still working on getting interrupted syscalls right (surprisingly difficult).

But the tick numbers for alarm2 look perfectly okay to me! That means the llwpcb approach appears to be working, at first glance.

rr record alarm fails with this message now

Oops, my mistake. Again, the tick numbers look fine, we just failed to re-enable interrupts when the CPU reported them disabled in the XSAVE area.

It'll probably be a while until I get the lwp-new branch to actually pass all the tests I care about, but at this point it looks to me like we should see identical behavior on your CPU and mine, so I can test here. I'd like to run one more test to confirm that, but I don't quite have it ready yet.

Thanks again for your patience, and sorry it took me so long.

pipcet commented 8 years ago

Okay, there's a new tag at https://github.com/pipcet/rr/tree/lwp-new-20160718; on this machine, it passes about 90% of the 64-bit tests, including quite a few that indicate LWP is working.

It'd be great to know if it passes any significant number of tests on your CPU. I'm building with gcc and 32-bit support disabled right now, and 32-bit support will need some work (but not too much).

Limitations right now are that the monkeypatcher is disabled, 32-bit support needs to be disabled in ccmake, plus a fair number of test failures to be investigated. (Some of which are WONTFIXes, for now: nested rr sessions don't work, and the code is currently using the stack while executing llwpcb). Plus the whole thing is probably much slower than it needs to be.

One thing I'm thinking about is to submit a minimal kernel patch that doesn't support LWP threshold interrupts: instead of using those, we can unmap the LWP buffer page, which results in a SIGSEGV when an event record is written, which we can catch and handle much like the rdtsc segfaults; we'd then setup LWP anew after returning from the signal handler.

edwintorok commented 8 years ago

On 2016-07-18 09:18, pipcet wrote:

Okay, there's a new tag at https://github.com/pipcet/rr/tree/lwp-new-20160718; on this machine, it passes about 90% of the 64-bit tests, including quite a few that indicate LWP is working.

Success! Thanks a lot for all the work!

You can find the logs in the usual place.

It'd be great to know if it passes any significant number of tests on your CPU. I'm building with gcc and 32-bit support disabled right now, and 32-bit support will need some work (but not too much).

I've also added the output of '(ulimit -t 15; ninja test 2>&1|tee testlog)', which has 795 tests that pass! I killed the testsuite around test 1464/1716 when the 32-bit tests started cause they would just hang forever, but like you say thats expected.

Limitations right now are that the monkeypatcher is disabled, 32-bit support needs to be disabled in |ccmake|, plus a fair number of test failures to be investigated. (Some of which are WONTFIXes, for now: nested rr sessions don't work, and the code is currently using the stack while executing llwpcb). Plus the whole thing is probably much slower than it needs to be.

Are there some flags I could pass to 'ninja test' that would kill the tests after some timeout? Then I could run all the tests and see exactly which fail. (I tried 'ulimit -t' but that only works if the failing test uses 100% CPU.)

One thing I'm thinking about is to submit a minimal kernel patch that doesn't support LWP threshold interrupts: instead of using those, we can unmap the LWP buffer page, which results in a SIGSEGV when an event record is written, which we can catch and handle much like the |rdtsc| segfaults; we'd then setup LWP anew after returning from the signal handler.

Is SIGSEGV handling faster than what you're using now? Is it possible to reliably differentiate between SIGSEGV from the app and due to LWP page?

pipcet commented 8 years ago

Success!

Excellent! The logs look good, too, as far as I can tell.

Are there some flags I could pass to 'ninja test' that would kill the tests after some timeout?

I'm curious about that as well. The documentation for ctest has a timeout option, but it doesn't appear to be working. What I'm doing here is

The last command runs 30 tests at once, which means one test every four seconds even if they just sit there and time out. It also weeds out all tests but those that fail consistently, which is okay right now since I'm going for the low-hanging fruit first and intermittent failures are harder :-)

Is SIGSEGV handling faster than what you're using now?

Probably not, and there are a lot of things we could tweak first to improve performance, which is indeed very bad right now. For example, if we had code to recover from overshoots we could drastically reduce the skid size (and also remove the cpupower hack, which does increase power consumption a lot), leading to much less single-stepping.

While catching SIGSEGV is unusual, page faults while running LWP are not, so I'm expecting that part of the hardware/microcode to be quite well-tested. But the main reason is it reduces the kernel patch to about ten lines. In fact I'm a bit surprised Linux doesn't allow root to add xfeatures on the fly, which would have been nice for testing.

Is it possible to reliably differentiate between SIGSEGV from the app and due to LWP page?

In practice, yes, since we have the fault address. In theory it's harder since the application might be accessing that address, but it really shouldn't.

pipcet commented 8 years ago

Okay, I've been busy. 32-bit support appears to be mostly working: https://github.com/pipcet/rr/tree/lwp-new-20160718-2

I would be very interested to know which tests fail for you, because the LWP code changed (slightly), which I hope did not break everything.

edwintorok commented 8 years ago

On 2016-07-18 21:40, pipcet wrote:

Okay, I've been busy. 32-bit support appears to be mostly working: https://github.com/pipcet/rr/tree/lwp-new-20160718-2

I would be very interested to know which tests fail for you, because the LWP code changed (slightly), which I hope did not break everything.

Thanks, 1144 passed, including some 32-bit tests (see updated testlog that I just pushed): The command I used: ctest --schedule-random -j30 https://gist.githubusercontent.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/490f59be92a73e6a5c4d974419d0c902a98d49ac/testlog

pipcet commented 8 years ago

Thanks! That is strange: you're seeing more failures than me (in particular, only 600 of the 64-bit tests passed, fewer than in the earlier log), but some tests that require LWP to actually be working are passing, so it's not that it's back to not working at all.

One reason might be that the test suite isn't tied to CPU 0, IIUC. Maybe the results would be better if that were changed. I'll try preparing something testable.

pipcet commented 8 years ago

Okay, I've pushed a new tag to https://github.com/pipcet/rr/tree/lwp-new-20160719; if you want to test more, can you try running ctest on that without the -j option? It passes about 90% of tests on the first try here, so you should be seeing similar results (otherwise, it would be great if you could run ctest --rerun-failed in a loop to see how many tests actually fail consistently).

edwintorok commented 8 years ago

On 2016-07-19 20:38, pipcet wrote:

Okay, I've pushed a new tag to https://github.com/pipcet/rr/tree/lwp-new-20160719; if you want to test more, can you try running ctest on that without the -j option? It passes about 90% of tests on the first try here, so you should be seeing similar results (otherwise, it would be great if you could run |ctest --rerun-failed| in a loop to see how many tests actually fail consistently).

Thanks, new logs pushed to usual place, I got around 96% tests pass, very nice! I see 1655/1716 passed on first run, and 1/61 passed on 1st rerun, 0/60 passed on 2nd rerun, so that looks like just 1 non-deterministic test:

1st run: Start 1357: wait_sigstop-32 1357/1716 Test #1357: wait_sigstop-32 ...........................................***Failed Error regular expression found in output. Regex=[FAILED] 0.21 sec rerun: Start 1357: wait_sigstop-32 45/61 Test #1357: wait_sigstop-32 ...................................... Passed 0.21 sec

pipcet commented 8 years ago

Not much more progress, I'm afraid; I'm down to 11 tests that fail consistently, but only if I try running two variants of my code that I don't fully understand—some tests pass only if I disable LWP more often, some pass only if I don't. I'll have to investigate that further, since I think it's only sigreturn (two 32-bit variants, one 64-bit variant) that should be exepected to damage the LWP state, so ideally we'd only disable LWP counting around sigreturn, and hopefully that'll get us down to a reasonable number of failures.

pipcet commented 8 years ago
100% tests passed, 0 tests failed out of 1716

Okay, there's a new tag at https://github.com/pipcet/rr/tree/lwp-new-20160725, which passes all tests for me (I think the record-replay test is disabled)! It turns out I can take a somewhat simpler approach than I did at first, which will make merging things easier: we don't need to keep track of whether we're in a syscall or returning from one, we can just unconditionally enable LWP whenever we're stopped.

One thing I haven't managed to replicate is the logic in compute_trap_reasons; the upstream version avoids looking at the siginfo structure for a SIGTRAP to determine whether it was caused by a single-step syscall or an external signal.

I also haven't quite convinced myself that we're doing the best possible thing wrt sigreturn, and I'm pretty sure we're not doing the right thing when set_extra_regs is called, but that doesn't appear to get tested properly.

edwintorok commented 8 years ago

On 2016-07-25 21:18, pipcet wrote:

|100% tests passed, 0 tests failed out of 1716 |

Okay, there's a new tag at https://github.com/pipcet/rr/tree/lwp-new-20160725, which passes all tests for me (I think the record-replay test is disabled)! It turns out I can take a somewhat simpler approach than I did at first, which will make merging things easier: we don't need to keep track of whether we're in a syscall or returning from one, we can just unconditionally enable LWP whenever we're stopped.

Nice job, we're getting close! it only fails 17 tests here:

The following tests FAILED: 189 - mmap_discontinuous (Failed) 190 - mmap_discontinuous-no-syscallbuf (Failed) 213 - mremap (Failed) 214 - mremap-no-syscallbuf (Failed) 369 - setuid (Failed) 370 - setuid-no-syscallbuf (Failed) 554 - checksum_sanity-no-syscallbuf (Failed) 737 - check_patched_pthread (Failed) 738 - check_patched_pthread-no-syscallbuf (Failed) 1047 - mmap_discontinuous-32 (Failed) 1048 - mmap_discontinuous-32-no-syscallbuf (Failed) 1071 - mremap-32 (Failed) 1072 - mremap-32-no-syscallbuf (Failed) 1227 - setuid-32 (Failed) 1228 - setuid-32-no-syscallbuf (Failed) 1595 - check_patched_pthread-32 (Failed) 1596 - check_patched_pthread-32-no-syscallbuf (Failed) Errors while running CTest

pipcet commented 8 years ago

I've observed that some of those tests fail for me if I set /proc/sys/kernel/randomize_va_space to 1 or 2, and succeed if it's set to 0. If you could use --output-on-failure when running ctest, we should be able to see what causes those failures, at least in some cases. (The tests create files in a /tmp/rr-test-TESTNAME-NONCE directory, and I'm not sure whether --output-on-failure includes the GDB logs).

I suspect setuid fails because of security settings, too, but I haven't seen a failure here.

I've investigated the intermittent failures here, and many of them are interrupt latency issues. I've experimented with an unmapped (well, mmapped-with-PROT_NONE) LWP buffer and catching the resulting segmentation faults, and so far it looks like that actually works with SKID_SIZE set to 3, which is a nice change.

edwintorok commented 8 years ago

On 2016-07-27 20:27, pipcet wrote:

I've observed that some of those tests fail for me if I set |/proc/sys/kernel/randomize_va_space| to 1 or 2, and succeed if it's set to 0. If you could use |--output-on-failure| when running |ctest|, we should be able to see what causes those failures, at least in some cases. (The tests create files in a |/tmp/rr-test-TESTNAME-NONCE| directory, and I'm not sure whether |--output-on-failure| includes the GDB logs).

I suspect setuid fails because of security settings, too, but I haven't seen a failure here.

I've accidentally run that test as root when it failed, running as normal user succeeds.

Running as a normal user and randomize_va_space set to 0 (it was 2 before) I get only these failures:

The following tests FAILED: 737 - check_patched_pthread (Failed) 738 - check_patched_pthread-no-syscallbuf (Failed) 1595 - check_patched_pthread-32 (Failed) 1596 - check_patched_pthread-32-no-syscallbuf (Failed)

See testlog3 in the repo

pipcet commented 8 years ago

Hmm. According to that log, elision_init is a symbol visible to gdb, but it isn't patched by patch_after_mmap. I don't think that's a problem on family 15h, as AMD has no lock elision features (yet), but it would be interesting to figure out what's going on there.

I imagine what might be happening is that elision_init is now declared static in libc, so its symbol isn't in the symtab, but it's still visible to gdb.

In any case, I don't think that will cause problems on existing CPUs.

rocallahan commented 8 years ago

I have /proc/sys/kernel/randomize_va_space set to 2 and all tests pass for me, so I don't know what's up with that, though it's probably not hard to figure out.

pipcet commented 8 years ago

Hmm. I really don't know what's up with that. I'm speculating here, but is it possible you've disabled address space on a per-process basis, with setarch -R or similar? Looking at src/test/mmap_discontinuous.c, that test looks like it should indeed fail with ASLR, and it fails for me without being run under rr at all.

rocallahan commented 8 years ago

Definitely not using setarch or anything like that.

On Linux, ASLR randomizes the positions of various things but, as I understand it, generally not the relative positions of mmaps of the same type. Certainly these tests pass for most users who've run them. Maybe you have a patched kernel that does more aggressive randomization?

rocallahan commented 8 years ago

Whatever the issue is, I'd be happy to accept PRs that let those tests bail out with a pass if their adjacency assumptions aren't met. Or better still, using some MAP_FIXED hackery to force those assumptions to hold.

edwintorok commented 8 years ago

On 2016-07-28 01:48, rocallahan wrote:

Definitely not using |setarch| or anything like that.

On Linux, ASLR randomizes the positions of various things but, as I understand it, generally /not/ the /relative/ positions of mmaps of the same type. Certainly these tests pass for most users who've run them. Maybe you have a patched kernel that does more aggressive randomization?

On 2016-07-28 01:43, pipcet wrote:

Hmm. I really don't know what's up with that. I'm speculating here, but is it possible you've disabled address space on a per-process basis, with |setarch -R| or similar? Looking at src/test/mmap_discontinuous.c, that test looks like it should indeed fail with ASLR, and it fails for me without being run under |rr| at all.

You're right mmap_discontinous fails without running under rr when randomize_va_space is set to 2, and passes when set to 0. I'm using gentoo-sources 4.6.4, and here's my .config: https://gist.githubusercontent.com/edwintorok/1b4d95b4cb0b79bd2d85e6f3dc457e73/raw/2ff7721a8b2202d84e1fb07a94cf24ffea865646/.config

With randomize_va_space set to 1 or 2 I get addresses and differences like this: 0x7f4c5778c000, 0x7f4c57786000, 24576, -24576 0x7f358f1f4000, 0x7f358f1ee000, 24576, -24576 0x7fdde81e4000, 0x7fdde81de000, 24576, -24576 0x7f8d72ae4000, 0x7f8d72ade000, 24576, -24576

The difference always seems to be 24576, and if I change the 2nd mmap's size the difference changes, so perhaps it uses different alignment when randomization is on?

pipcet commented 8 years ago

It's entirely possible it's a kernel configuration option; my kernel is patched, in order to use LWP, but not beyond that.

pipcet commented 8 years ago

The difference always seems to be 24576

Thanks for investigating that. That means I was definitely wrong about this being merely the result of ASLR, since it would be random then.

I've pushed a new tag to https://github.com/pipcet/rr/tree/lwp-new-20160730; it uses the SIGSEGV approach, which I found improves latency here, so that opens the possibility it breaks everything on other CPUs again; testing would be very much appreciated. Ideally, this version should work both with the test suite and in practice.

Most of my modifications have moved to a new file, LWP.cc; the generic code has changed to use an abstract TicksSource base class rather than PerfCounters, which I think is the most sensible way of doing things: we're performing system calls, so the extra overhead of a virtualized call should be negligible.

There's one detail that I'm really unhappy with: when we divert execution to our llwpcb instruction, then get a signal, we will have performed the waitpid call for that signal by the time we find out. But the caller expects to do another wait and get the signal status, so right now I'm using an extra flag in the Task class to remember such premature stops: https://github.com/pipcet/rr/commit/a43efef2f05efc28edad85141d144caa5482d076#diff-b509a7939392c11bb3c517b00da4526fR1089

Can anyone think of a better way of handling this? It's not clear to me that stashing the signal is always a valid thing to do.

pipcet commented 8 years ago

I'm having trouble running rr under a very recent checkout of the Linux kernel. I suspect it's due to https://github.com/torvalds/linux/commit/7a1e8b80fb1e8ead4cec15d1fc494ed290e4d2e9. Changelog entry:

  • Seccomp: remove 2-phase API, close hole where ptrace can change syscall #

I don't want to open a new issue for this as I don't have an Intel CPU to confirm.