blackcon commented 2 years ago

1. install kernel as hAFL2

2. install qemu6.0.0

3. install WindowsOS as level0 on QEMU

Running Command QEMU

./hAFL2/qemu-6.0.0/build/x86_64-softmmu/qemu-system-x86_64 -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+intel-pt,-hypervisor,+vmx -machine q35 -enable-kvm -m 6144 -hda ./windows.qcow2 -bios /root/hAFL2/OVMF_CODE-pure-efi.fd -usbdevice tablet
Setup Hyper-V on Level0
Install Windows OS as level1 on Level0

NOTE: Network should connect Network.

4. Setup Level2 VM within Level1

Turn off Level2 and Execute command using powershell within Level1

Connect-VMNetworkAdapter -VMName level2 -SwitchName "Default Switch" Set-VMFirmare level2 -EnableSecureBoot off
Run the following command from within PowerShell:

(Get-NetAdapter)[0].InterfaceDescription
- Modify the ourName variable within the CPHarness driver to the output of the previous command,
- for example, if the output was Microsoft Hyper-V Network Adapter, assign the variable with:
  
  UNICODE_STRING ourName = RTL_CONSTANT_STRING(L"Microsoft Hyper-V Network Adapter");
Turn on Level1 and Configuring bcdedit.exe whtin Level2

bcdedit /set testsigning on bcdedit /set nointegritychecks on bcdedit -set loadoptions DDISABLE_INTEGRITY_CHECKS

5. Compile Necessary Binaries

Compile hAFL2’s fuzzing binaries by executing the following within bash:

pushd ./hAFL2/targets/windows_x86_64 && ./compile.sh && popd
Use Visual Studio to compile both drivers from within the hAFL2\drivers folder.
- CPHarness (Child Partition Harness) - this driver will be installed within the child partition VM, and will send packets of fuzzing payloads to the root partition's VMSwitch.
- CrashMonitoring - this driver will send root partition crashes to hAFL2 by using an hypercall interface.
Within VS:
- right click on each project -> Properties -> Driver Signing -> General
- Change "Sign Mode" to "Test Sign"
- Set "File Digest Algorithm" to "SHA1"
- Then compile.
  6. Optimizing the Crash Monitoring and Disabling DSE for Root Partition VM
Execute the Root Partition VM:~

~./hAFL2/qemu-6.0.0/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+intel-pt,-hypervisor,+vmx -usbdevice tablet -m 16384 -bios /root/hAFL2/OVMF_CODE-pure-efi.fd -drive file=windows.qcow2 -machine q35 -net none
In order to make the crash monitoring functionality operate faster, open PowerShell (within the root partition VM) as an Administrator and execute the following command:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl" -Name "CrashDumpEnabled" -Value 0
Disable Driver Signature Enforcement from within an elevated command prompt (Restart the root partition VM once you're done):

bcdedit /set testsigning on && bcdedit /set nointegritychecks on && bcdedit -set loadoptions DDISABLE_INTEGRITY_CHECKS

7. Preparing the Root and Child Partition VMs for Fuzzing

Enable Driver Verifier for vmswitch.sys on the Root Partition VM:

blackcon commented 2 years ago

Create a Fuzzing Snapshot

Create a VM overlay in a dedicated overlays folder: (Use the absolute path of windows.qcow2!)
./hAFL2/qemu-6.0.0/build/qemu-img create -f qcow2 -b windows.qcow2 overlay_0.qcow2

Run the VM overlay:
./hAFL2/qemu-6.0.0/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+intel-pt,-hypervisor,+vmx -usbdevice tablet -m 16384 -bios /root/hAFL2/OVMF_CODE-pure-efi.fd -drive file=overlay_0.qcow2 -machine q35

1. Retrieving VMSwitch.sys address range (Root Partition VM)

Open a command prompt as an Administrator from within the root partition VM.
Execute C:\info.exe as an Administrator.
Check that hAFL2 server now has the /tmp/kAFL_info.txt file, and that it contains a loaded modules list of the root partition VM, with loaded address range for each modules. (Copy this file, you'll use it soon.)

2. Installing Crash Monitoring Driver (Root Partition VM)

Install the Crash Monitoring Driver on the Root Partition VM:
- Execute OSRLoader.exe
- Change the Driver Path to C:\CrashMonitoringDriver.sys
- Click on "Register Service"
- Click on "Start Service"
- You may close the window now.

3. Installing Child Partition Harness Driver (Child Partition VM)

Within the Child Partition VM, open dbgview64.exe as an Administrator.

Click on the Capture Menu, then click on Capture Kernel.
Open a command prompt as an Administrator and install the CPHarness driver on the Child Partition VM by using devcon.exe (Approve the pop-up.)

C:\devcon.exe install CPHarness.inf root\CPHarness
Check DbgView window, and look for the following log lines:

PoolVNC is OK + Channel is OK: [ADDRESS]
Note

If you don't see these log lines, you need to make sure you modified the ourName variable within the CPHarness driver according to the NDIS.sys offset of the Child Partition VM as stated before (explained in the one of the previous section.)

Occured issue
- Can not Find Miniport Channel
- Trouble shooting ~- Settings DEBUGPORT using bcdedit.exe for debugging to kernel~
  ~bcdedit /dbgsettings serial DEBUGPORT:2 BAUDRATE:115200~ ~bcdedit /debug on~
  - Just Update Windows least Versin, and should modify miniportChannel offset at CPHarness sourcecode (and build)

4. Creating the snapshot (Final Step)

Within the Child Partition VM, open a command prompt as an Administrator and execute C:\loader.exe

This will create a snapshot to which the fuzzer will return after crashes.

Execute C:loader.exe whithin Child Partition VM
And than turn off QEMU automatically.
Optional: Duplicating VM overlays Make sure all of the overlays files are within the same overlays folder and execute the following command:

for f in overlay_{1..X}.qcow2; do cp overlay_0.qcow2 $f; done

blackcon commented 2 years ago

Test communication KVM(L0) and L1/L2

Get Device Address From kAFL Server(L0)

Command

python3 /root/hAFL2/kAFL-Fuzzer/kafl_info.py \
-work_dir /root/work \
-vm_dir /root/hAFL_qcow2_dir \
-bios /root/hAFL2/OVMF_CODE-pure-efi.fd \
-mem 16384 \
-agent hAFL2/targets/windows_x86_64/bin/info/info.exe \
-v

result

blackcon commented 2 years ago

Modified CPHarness for Virtual Storage

0. Looking for kmclChannelListLocation at DllInitialize() in vmbkcl.sys

How to find kmclChannelListLocation address?

CASE1: Looking for first existing "lea rax, SOMETHING" code snippet, SOMETHING should be pointing at KmclChannelList (build 19044.1526)

 ptr = (PUINT8)KernelGetProcAddress(vmbkclBaseAddress, (PCHAR)&vmbDllInitialize);
 DbgPrintEx(DPFLTR_IHVDRIVER_ID, DEBUG_LEVEL, "CPHarness: DllInitialize Address: %p\n", ptr);
 if (ptr)
 {
     //Looking for first existing "lea     rax, SOMETHING" code snippet, SOMETHING should be pointing at KmclChannelList
     for (int x = 0; x < 0x200; x++)
     {
         ptr++;
         if (ptr[0] == 0x48 && ptr[1] == 0x8d && ptr[2] == 0x05 && ptr[3] == 0x3c && ptr[4] == 0x90 && ptr[5] == 0xff && ptr[6] == 0xff)     // (vmbkmcl.sys in guest)
         {
             kmclChannelListLocation = (PVOID)((UINT64)ptr + 0xFFFFFFFF00000007 + *((PUINT32)(ptr + 3)));
             DbgPrintEx(DPFLTR_IHVDRIVER_ID, DEBUG_LEVEL, "CPHarness: kmclChannelListLocation: %p\n", kmclChannelListLocation);
             break;
         }
     }
 }

CASE2: Looking for structure WPP_MAIN_CB based offset. (note: It may be different for each OS version.)

LONG WPP_MAIN_CB_OFFSET = 0x130E0; //  vmbkmcl.sys build 10.0.19044.1526, offset 0x130E0 ==> vmbkmcl!WPP_MAIN_CB 
if (vmbkclBaseAddress)
{
    //kmclChannelListLocation =poi(vmbkmcl+0x130e0+WPP_MAIN_CB_OFFSET)+0x20 
    kmclChannelListLocation = (PVOID)((UINT64)vmbkclBaseAddress + WPP_MAIN_CB_OFFSET);   // vmbkmcl!WPP_MAIN_CB
    kmclChannelListLocation = (PVOID)((UINT64)kmclChannelListLocation + 0xA0);          // vmbkmcl!WPP_MAIN_CB.DeviceQueue
    kmclChannelListLocation = (PVOID)(*(PUINT64)kmclChannelListLocation + 0x20);        // vmbkmcl!WPP_MAIN_CB.DeviceQueue.kmclChannelListLocation
}

1. Testing Load the Driver(CPHarness) on Hyper-v `without hAFL2`

2. Testing Load the Driver(CPHarness) on Hyper-v `whithin hAFL2`

Load Driver
Load Driver (Clean Log)

blackcon commented 2 years ago

Sanity Check

Execute fuzzer

Command

python3 /root/hAFL2/kAFL-Fuzzer/kafl_fuzz.py \
-work_dir ./work \
--purge \
-vm_dir /root/hAFL_qcow2_dir \
-bios ./hAFL2/OVMF_CODE-pure-efi.fd \
-mem 16384 \
-agent /root/hAFL2/targets/windows_x86_64/bin/fuzzer/packet_sender.exe\
-p 1 \
-ip0 0xfffff80749d10000-0xfffff8054cb52000 \
--debug \
-v \
-seed_dir /root/hAFL2/kAFL-Fuzzer/seed/hv/

Image

View Log
- Command: tail -f work/debug.log
- Image

Display kAFL-GUI

Command

python3 hAFL2/kAFL-Fuzzer/kafl_gui.py ~/work

Image

blackcon commented 2 years ago

Sanity Check

1. Execute fuzzer

   * Command
     ```
     python3 /root/hAFL2/kAFL-Fuzzer/kafl_fuzz.py \
      -work_dir ./work \
      --purge \
      -vm_dir /root/hAFL_qcow2_dir \
      -bios ./hAFL2/OVMF_CODE-pure-efi.fd \
      -mem 16384 \
      -agent /root/hAFL2/targets/windows_x86_64/bin/fuzzer/packet_sender.exe\
      -p 1 \
      -ip0 0xfffff80749d10000-0xfffff8054cb52000 \
      --debug \
      -v \
      -seed_dir /root/hAFL2/kAFL-Fuzzer/seed/hv/ 
     ```
   * Image
     ![image](https://user-images.githubusercontent.com/6852711/156588482-000d40e7-92a4-40a1-b69f-88cc11745c5a.png)

2. View Log

   * Command: `tail -f work/debug.log`
   * Image
     ![image](https://user-images.githubusercontent.com/6852711/156588608-f3f26e6d-84cf-4647-bec9-11f23bc1b11b.png)

3. Display kAFL-GUI

   * Command
     ```
     python3 hAFL2/kAFL-Fuzzer/kafl_gui.py ~/work
     ```
   * Image
     ![image](https://user-images.githubusercontent.com/6852711/156588407-3bfee4ed-74f0-42f2-be4a-6fb541576ac6.png)

What the..

blackcon commented 2 years ago

Sanity Check

What the..

Trouble Shooting..

Modyfied Packet Size for target driver at CPHarness\Driver.c

As-Is

UINT NVSP_RNDIS_PKT_SIZE = 0x28;
pNvspRndisPkt = ExAllocatePool2(POOL_FLAG_NON_PAGED, NVSP_RNDIS_PKT_SIZE , HARNESS_POOL_TAG);  // for NDIS.sys
...
status = SendPacket(pNvspRndisPkt, (UINT32)NVSP_RNDIS_PKT_SIZE , pFuzzPayload, (ULONG)fuzzPayloadSize);

To-Be

UINT TARGET_PKT_SIZE = 0x40;
pNvspRndisPkt = ExAllocatePool2(POOL_FLAG_NON_PAGED, TARGET_PKT_SIZE  , HARNESS_POOL_TAG);  // for TARGET.sys
...
status = SendPacket(pNvspRndisPkt, (UINT32)TARGET_PKT_SIZE , pFuzzPayload, (ULONG)fuzzPayloadSize);

Sanity Check After Trouble Shootings

Image

Run kAFL.py

View Log tail -f ~/work/debug.log
View kAFL-GUI
Some Problems..
Occurred Disk Full (See: /dev/sdb6)

Tracing


root@hafl-System-Product-Name:/# du -sh /var
173G /var
root@hafl-System-Product-Name:/var# du -sh /var/log/
170G /var/log/
root@hafl-System-Product-Name:/var/log# ll -SS
total 173792960
-rw-r-----   1 syslog            adm             88837206016  3월  8 17:27 syslog
-rw-r-----   1 syslog            adm             88827826176  3월  8 17:06 kern.log
-rw-r-----   1 syslog            adm               277410057  3월  8 02:21 kern.log.1

Delete syslog, kern.log

root@hafl-System-Product-Name:/home/hafl# df -l Filesystem 1K-blocks Used Available Use% Mounted on udev 32859736 0 32859736 0% /dev tmpfs 6578420 1968 6576452 1% /run /dev/sdb6 199523664 15565984 173752752 9% /

blackcon commented 2 years ago

Finally Starting Fuzz..

Desktop Image
WinVer
kAFL hprintf Log
kAFL Debug LOG

blackcon / HVFUZZ

SETUP #1

1. install kernel as hAFL2

2. install qemu6.0.0

3. install WindowsOS as level0 on QEMU

4. Setup Level2 VM within Level1

5. Compile Necessary Binaries

6. Optimizing the Crash Monitoring and Disabling DSE for Root Partition VM

7. Preparing the Root and Child Partition VMs for Fuzzing

Create a Fuzzing Snapshot

1. Retrieving VMSwitch.sys address range (Root Partition VM)

2. Installing Crash Monitoring Driver (Root Partition VM)

3. Installing Child Partition Harness Driver (Child Partition VM)

4. Creating the snapshot (Final Step)

Test communication KVM(L0) and L1/L2

Modified CPHarness for Virtual Storage

0. Looking for kmclChannelListLocation at DllInitialize() in vmbkcl.sys

How to find kmclChannelListLocation address?

1. Testing Load the Driver(CPHarness) on Hyper-v `without hAFL2`

2. Testing Load the Driver(CPHarness) on Hyper-v `whithin hAFL2`

Sanity Check

Sanity Check

Sanity Check

Trouble Shooting..

Sanity Check After Trouble Shootings

Image

Some Problems..

Delete syslog, kern.log

Finally Starting Fuzz..

blackcon / HVFUZZ

SETUP #1

1. install kernel as hAFL2

2. install qemu6.0.0

3. install WindowsOS as level0 on QEMU

4. Setup Level2 VM within Level1

5. Compile Necessary Binaries

6. Optimizing the Crash Monitoring and Disabling DSE for Root Partition VM

7. Preparing the Root and Child Partition VMs for Fuzzing

Create a Fuzzing Snapshot

1. Retrieving VMSwitch.sys address range (Root Partition VM)

2. Installing Crash Monitoring Driver (Root Partition VM)

3. Installing Child Partition Harness Driver (Child Partition VM)

4. Creating the snapshot (Final Step)

Test communication KVM(L0) and L1/L2

Modified CPHarness for Virtual Storage

0. Looking for kmclChannelListLocation at DllInitialize() in vmbkcl.sys

How to find kmclChannelListLocation address?

1. Testing Load the Driver(CPHarness) on Hyper-v without hAFL2

2. Testing Load the Driver(CPHarness) on Hyper-v whithin hAFL2

Sanity Check

Sanity Check

Sanity Check

Trouble Shooting..

Sanity Check After Trouble Shootings

Image

Some Problems..

Delete syslog, kern.log

Finally Starting Fuzz..

1. Testing Load the Driver(CPHarness) on Hyper-v `without hAFL2`

2. Testing Load the Driver(CPHarness) on Hyper-v `whithin hAFL2`