search

IntelLabs / kAFL

A fuzzer for full VM kernel/driver targets
https://intellabs.github.io/kAFL/
MIT License
625 stars 88 forks source link

"[QEMU-NYX] Waiting for snapshot to start fuzzing..." but nothing #283

Open leozhao2333 opened 1 week ago

leozhao2333 commented 1 week ago

Hello! I have some issues when using kafl to fuzz windows .sys file. It seems that QEMU-NYX can't booting VM to starting fuzzing. The terminal output will stuck in "[QEMU-NYX] Waiting for snapshot to start fuzzing..." for minute. The output log shows below, sorry for some confusion about personal information in ``

(.venv) *:~/Desktop/kafl/kAFL/kafl/examples/windows_x86_64$ kafl fuzz -p 32

    __                        __  ___    ________
   / /_____  _________  ___  / / /   |  / ____/ /
  / //_/ _ \/ ___/ __ \/ _ \/ / / /| | / /_  / /
 / ,< /  __/ /  / / / /  __/ / / ___ |/ __/ / /___
/_/|_|\___/_/  /_/ /_/\___/_/ /_/  |_/_/   /_____/
===================================================

<< kAFL Fuzzer >>

Warning: Launching without --seed-dir?
No PT trace region defined.
Warning: Requested 32 workers but 0 out of 32 vCPUs seem busy?
00:00:00:     0 exec/s,    0 edges,  0% favs pending, findings: <0, 0, 0>
Worker-00 Launching virtual machine...
*/Desktop/kafl/kAFL/kafl/qemu/x86_64-softmmu/qemu-system-x86_64
    -enable-kvm
    -machine kAFL64-v1
    -cpu kAFL64-Hypervisor-v1,+vmx
    -no-reboot
    -net none
    -display none
    -chardev socket,server,id=nyx_socket,path=/dev/shm/*/interface_0
    -device nyx,chardev=nyx_socket,workdir=/dev/shm/*,worker_id=0,bitmap_size=65536,input_buffer_size=131072
    -device isa-serial,chardev=kafl_serial
    -chardev file,id=kafl_serial,mux=on,path=/dev/shm/*/serial_00.log
    -m 4096
    -drive file=/home/*/.local/share/libvirt/images/windows_x86_64_vagrant-kafl-windows.img
    -monitor unix:/tmp/monitor.sock,server,nowait
    -fast_vm_reload path=/dev/shm/*/snapshot/,load=off
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Booting VM to start fuzzing...
Worker-01 Launching virtual machine...
Worker-02 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-03 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-04 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-05 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-06 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-07 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-08 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-09 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-10 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-11 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-12 Launching virtual machine...
Worker-13 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-14 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-15 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-16 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-17 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-18 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-19 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-20 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-21 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-22 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-23 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-24 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-25 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-26 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-27 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-28 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-29 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-30 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
Worker-31 Launching virtual machine...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-NYX] Warning: Invalid sharedir...
[QEMU-NYX] Waiting for snapshot to start fuzzing...

My core version is :

*:~$ uname -r
5.10.75-051075-generic

And my process information is:

*:~$ lscpu
架构：                   x86_64
  CPU 运行模式：         32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  字节序：               Little Endian
CPU:                     32
  在线 CPU 列表：        0-31
厂商 ID：                GenuineIntel
  型号名称：             13th Gen Intel(R) Core(TM) i9-13900HX
    CPU 系列：           6
    型号：               183
    每个核的线程数：     2
    每个座的核数：       24
    座：                 1
    步进：               1
    CPU 最大 MHz：       6900.0000
    CPU 最小 MHz：       800.0000
    BogoMIPS：           4838.40
    标记：               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
                         a cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss 
                         ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art
                          arch_perfmon pebs bts rep_good nopl xtopology nonstop_
                         tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes6
                         4 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr p
                         dcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline
                         _timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefe
                         tch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp
                          ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ep
                         t_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpc
                         id rdseed adx smap clflushopt clwb intel_pt sha_ni xsav
                         eopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp 
                         hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip pku 
                         ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir
                         64b fsrm md_clear serialize arch_lbr flush_l1d arch_cap
                         abilities
Virtualization features: 
  虚拟化：               VT-x
Caches (sum of all):     
  L1d:                   896 KiB (24 instances)
  L1i:                   1.3 MiB (24 instances)
  L2:                    32 MiB (12 instances)
  L3:                    36 MiB (1 instance)
NUMA:                    
  NUMA 节点：            1
  NUMA 节点0 CPU：       0-31
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy ba
                         rriers only; no swapgs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled
  Srbds:                 Not affected
  Tsx async abort:       Not affected
leozhao2333 commented 1 week ago

Sorry for the terminal output in Chinese :no_mouth:

5angjun commented 1 week ago

You have to change your kernel version to nyx kernel. Then you can enjoy kAFL

leozhao2333 commented 1 week ago

Hello! I do not install the Nyx kernal, but I actually install the NYX-QEMU patch, and I actually successfully started the KAFL before, but after once start up my machine, the components output becomes here 🥹

5angjun commented 1 week ago

It is different between nyx-qemu and nyx-kernel.

You can get nyx-qemu's code coverage if you patch your kernel to nyx kernel modifying hypervisor modules to get qemu coverage instead of host coverage.

leozhao2333 commented 1 week ago

Okey I see. I try to change the kernal to nyx-kernal, but do you have any suggestion about how to fix [QEMU-NYX] Booting VM to start fuzzing... but waiting and waiting and get no response Since I config some similar scripts using the kAFL successfully before, but suddenly just once start up my machine, it just into a while true do: waiting infinity loop

5angjun commented 1 week ago

I cannot fully understand your environment and Bugs. Because i don't know your full context.

But the way to resolve these kind of porblems is the set --debug options and observe the hprintf log or other log files from QEMU. Or reset the host Ubuntu and try reinstalling kAFL.

leozhao2333 commented 1 week ago

Sorry for my not precise expression. I meet the trouble is that: the kAFL sometimes work, but sometimes do not work even I don't change any config file. The cannot work behavior is: the first thread stuck in [QEMU-NYX] Booting VM to start fuzzing... and the other thread stuck in [QEMU-NYX] Waiting for snapshot to start fuzzing... BUT it in small probability randomly works, the attempts I have tried:

I wander how to fix this situation, if you need more information about my host system, could you please explicate what you need.

5angjun commented 15 hours ago

Can you show your agent.cpp or harness? Specifically, Insert hprintf line by line in your harness and observe the output. Through this, We can finally notice which point the fuzz was stucked with and point out what we need to do.