search

IntelLabs / kafl.linux

Linux kernel branches for confidential compute research
16 stars 6 forks source link

Cannot allocate main ToPA buffer! #4

Open Wenzel opened 1 year ago

Wenzel commented 1 year ago

Sometimes kAFL workers will raised an assertion failure message in QEMU: https://github.com/IntelLabs/kafl.qemu/blob/kafl_stable/nyx/pt.c#L327

This is due to an failed allocation in the linux kernel, and looking at dmesg logs, a few messages like Cannot allocate main ToPA buffer! appears.

These messages originates from here: https://github.com/IntelLabs/kafl.linux/blob/kvm-nyx-5.10.73/arch/x86/kvm/vmx/vmx_pt.c#LL775C15-L775C48

A possible fix is to drop your kernel file cache:

sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

The flag __GFP_NOWARN might be in question for this allocation failure. See also kernel memory allocation guide

il-steffen commented 1 year ago

Candidate fixes: https://github.com/nyx-fuzz/KVM-Nyx/pull/4 - report allocation failure to Qemu - merged https://github.com/nyx-fuzz/QEMU-Nyx/pull/43 - do not attempt mmap() on allocaiton failure - merged https://github.com/nyx-fuzz/KVM-Nyx/pull/5 - try harder to allocate - merged in sdv branch + nyx-6.0 candidate (917352fa9a9)

Wenzel commented 1 year ago

Getting ToPA buffer full issues on Nyx kernel 6.0 And a KVM crash originating from topa_full() function in vmx_pt.c

[336740.279414] [KVM-NYX] Error:        Cannot allocate main ToPA buffer!
[336740.279421] [KVM-NYX] Info: ToPA setup failed...
[336740.282401] [KVM-NYX] Error:        Cannot allocate main ToPA buffer!
[336740.282408] [KVM-NYX] Info: ToPA setup failed...
[336740.284820] [KVM-NYX] Error:        Cannot allocate main ToPA buffer!
[336740.284825] [KVM-NYX] Info: ToPA setup failed...
[336740.286784] [KVM-NYX] Error:        Cannot allocate main ToPA buffer!
[336740.286789] [KVM-NYX] Info: ToPA setup failed...
[336740.908527] BUG: kernel NULL pointer dereference, address: 0000000000000098
[336740.908535] #PF: supervisor read access in kernel mode
[336740.908538] #PF: error_code(0x0000) - not-present page
[336740.908541] PGD 0 P4D 0
[336740.908546] Oops: 0000 [#4] PREEMPT SMP PTI
[336740.908550] CPU: 0 PID: 272377 Comm: CPU 0/KVM Tainted: G      D   I        6.0.0-nyx+ #1
[336740.908555] Hardware name:  /NUC6i5SYB, BIOS SYSKLi35.86A.0073.2020.0909.1625 09/09/2020
[336740.908558] RIP: 0010:topa_full+0x6/0x60 [kvm_intel]
[336740.908581] Code: fc 5b 41 5c 5d c3 cc cc cc cc 65 8b 15 fb 36 34 3f 48 c7 c7 a5 cb ce c0 e8 ab 69 68 fc eb 90 0f 1f 44 00 00 0f 1f 44 00 00 55 <48> 8b 87 98 00 00 00 48 89 e5 a9 80 ff ff ff 75 1d 48 c1 e8 20 48
[336740.908586] RSP: 0018:ffffbc5a88043c70 EFLAGS: 00010046
[336740.908590] RAX: 0000000000000000 RBX: ffff90e10d6d8000 RCX: ffff90e100d849c0
[336740.908594] RDX: ffff90e1f7040500 RSI: ffffffffc0c21f4f RDI: 0000000000000000
[336740.908597] RBP: ffffbc5a88043cc0 R08: ffff90e46ec00000 R09: 00000000000000b8
[336740.908600] R10: 0000000000000007 R11: 00007fe251e5c000 R12: ffff90e161155180
[336740.908603] R13: 0000000000000000 R14: 0000000000000000 R15: ffff90e161155398
[336740.908607] FS:  00007fe24cf32700(0000) GS:ffff90e46ec00000(0000) knlGS:0000000000000000
[336740.908610] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[336740.908614] CR2: 0000000000000098 CR3: 000000012c498002 CR4: 00000000003726f0
[336740.908617] Call Trace:
[336740.908620]  <TASK>
[336740.908624]  ? vmx_vcpu_run+0x32/0x1470 [kvm_intel]
[336740.908645]  kvm_arch_vcpu_ioctl_run+0x8f3/0x1780 [kvm]
[336740.908739]  ? vcpu_put+0x4a/0x70 [kvm]
[336740.908814]  kvm_vcpu_ioctl+0x29b/0x6f0 [kvm]
[336740.908886]  ? __fget_light+0xa7/0x130
[336740.908894]  __x64_sys_ioctl+0x92/0xd0
[336740.908901]  do_syscall_64+0x59/0x90
[336740.908906]  ? exit_to_user_mode_prepare+0x49/0x1a0
[336740.908913]  ? syscall_exit_to_user_mode+0x26/0x50
[336740.908918]  ? do_syscall_64+0x69/0x90
[336740.908923]  ? fpregs_assert_state_consistent+0x2a/0x50
[336740.908929]  ? exit_to_user_mode_prepare+0x49/0x1a0
[336740.908934]  ? syscall_exit_to_user_mode+0x26/0x50
[336740.908939]  ? do_syscall_64+0x69/0x90
[336740.908943]  ? irqentry_exit+0x3b/0x50
[336740.908947]  ? sysvec_reschedule_ipi+0x85/0x130
[336740.908952]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[336740.908957] RIP: 0033:0x7fe2511423ab
[336740.908961] Code: 0f 1e fa 48 8b 05 e5 7a 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b5 7a 0d 00 f7 d8 64 89 01 48
[336740.908964] RSP: 002b:00007fe24cf315b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[336740.908969] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fe2511423ab
[336740.908972] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000014
[336740.908976] RBP: 000055d1523e12f0 R08: 000055d14fd0f270 R09: 00007fe2380040b8
[336740.908979] R10: 00007fe2380050a0 R11: 0000000000000246 R12: 0000000000000000
[336740.908981] R13: 000055d15037c080 R14: 00007fff7b2045c0 R15: 00007fe24cf31880
[336740.908989]  </TASK>
[336740.908991] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_conntrack_netlink xfrm_user xfrm_algo rfcomm xt_addrtype br_netfilter nf_tables nfnetlink bridge stp llc overlay ip6table_filter ip6table_nat ip6table_mangle ip6_tables ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog xt_conntrack cmac algif_hash algif_skcipher iptable_filter af_alg bnep snd_soc_skl intel_rapl_msr snd_soc_hdac_hda snd_hda_ext_core mei_hdcp snd_soc_sst_ipc snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi intel_rapl_common snd_hda_codec_hdmi snd_soc_core snd_compress xt_MASQUERADE intel_tcc_cooling iwlmvm snd_hda_codec_realtek ac97_bus x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp ledtrig_audio mac80211 snd_pcm_dmaengine coretemp libarc4 snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hda_codec iptable_nat snd_hda_core btrtl iwlwifi nf_nat btbcm rapl snd_hwdep intel_cstate btintel nf_conntrack nf_defrag_ipv6 snd_pcm btmtk snd_timer snd nf_defrag_ipv4 iTCO_wdt bluetooth ee1004
[336740.909085]  cfg80211 intel_pmc_bxt iTCO_vendor_support soundcore ecdh_generic 8250_dw ecc intel_xhci_usb_role_switch mei_me intel_pch_thermal mei ir_rc6_decoder rc_rc6_mce xt_CHECKSUM ite_cir acpi_pad mac_hid xt_tcpudp iptable_mangle kvm_intel kvm binfmt_misc sch_fq_codel ramoops msr reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear i915 drm_buddy i2c_algo_bit ttm drm_display_helper cec crct10dif_pclmul crc32_pclmul rc_core sdhci_pci ghash_clmulni_intel drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt crypto_simd ahci fb_sys_fops cqhci intel_lpss_pci cryptd i2c_i801 drm intel_lpss xhci_pci e1000e i2c_smbus libahci idma64 sdhci xhci_pci_renesas video pinctrl_sunrisepoint
[336740.909183] CR2: 0000000000000098
[336740.909186] ---[ end trace 0000000000000000 ]---
[336740.909189] RIP: 0010:topa_full+0x6/0x60 [kvm_intel]
[336740.909210] Code: fc 5b 41 5c 5d c3 cc cc cc cc 65 8b 15 fb 36 34 3f 48 c7 c7 a5 cb ce c0 e8 ab 69 68 fc eb 90 0f 1f 44 00 00 0f 1f 44 00 00 55 <48> 8b 87 98 00 00 00 48 89 e5 a9 80 ff ff ff 75 1d 48 c1 e8 20 48
[336740.909213] RSP: 0018:ffffbc5a847c3c50 EFLAGS: 00010046
[336740.909217] RAX: 0000000000000000 RBX: ffff90e114577000 RCX: ffff90e1b20e2fc0
[336740.909220] RDX: ffff90e117df0100 RSI: ffffffffc0c21f4f RDI: 0000000000000000
[336740.909223] RBP: ffffbc5a847c3ca0 R08: 0000000016a7edac R09: 00000000001b775e
[336740.909226] R10: 00000000001b775e R11: fffddb3e539b56fc R12: ffff90e115fea8c0
[336740.909229] R13: 0000000000000000 R14: 0000000000000000 R15: ffff90e115fea8f8
[336740.909232] FS:  00007fe24cf32700(0000) GS:ffff90e46ec00000(0000) knlGS:0000000000000000
[336740.909236] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[336740.909239] CR2: 0000000000000098 CR3: 000000012c498002 CR4: 00000000003726f0