Tracing for function calls, printk, etc?

[Matheus-Garbelini]

Hi, is it possible to customise a hooking script to be able to print the symbols of functions call instructions?

Currently, Hoedur only support APIs to hook on basic blocks, interrupts, mmio read/write, ram read/write, but not function call.

Perhaps a on_function_call would enable us to quickly debug the firmware emulation via printing the stack trace. This would also enable to hooking a call to printk to show what the firmware is intending to log in real time.

[SWW13]

There is a mostly undocumented "debug hooks" feature for exactly your use-case.

Config example: https://github.com/fuzzware-fuzzer/hoedur-experiments/blob/2babc78b72b1331b3122c11f170e9367132b8aa2/targets/arm/Hoedur/loramac-node/CVE-2022-39274/config.yml#L34-L36

For more config details you can look at the internal config struct: https://github.com/fuzzware-fuzzer/hoedur/blob/main/emulator/src/hooks/debug.rs#L12-L21

For building you own tracing with scripts you can look at our implementation (trace::* functions can also be used in all scripts): https://github.com/fuzzware-fuzzer/hoedur/blob/main/emulator/hooks/modules/trace.rn

[Matheus-Garbelini]

@SWW13 Thanks a lot. Unfortunately, it seems a bit difficult for hadur to pass beyond Bluetooth Initialization errors check with zephyr OS. I mostly get Bluetooth Initialisation failed (error code -33) during firmware emulation (see below). Is there any optimisation, configuration or procedure to that allows us to reach Bluetooth initialisation so we can fuzz HCI inputs with Hoedur? Or eventually it will find the correct mmio inputs to proceed this BT firmware control flow?

[SWW13]

Without analyzing the firmware and fuzzing progress this is hard to generally answer.

Some things you can look into:

• Interrupt config: use interval: false (disables interrupt injection every n-th basic blocks, will only work if the firmware contains a reachable wfi) and mode: fuzzed (which gives the fuzzer control over which interrupt will be raised)
• Check the coverage to see where the fuzzer is stuck and either patch the firmware (source / binary) or if related to an MMIO read provide a hand crafted MMIO model.
• The Fuzzer may take multiple hours to days depending on the firmware complexity and hurdles, periodically check the fuzzer statistics in the log file to see if the edge count is increasing eventually or if the fuzzer is stuck.

[Matheus-Garbelini]

@SWW13 Thanks.

Turns out hoedur had some issues triggering an specific ISR, which was needed to fill the pool of random numbers.

For example, calls to rng_pool_get expected a return greater than 0 to exit a infinite while loop in the caller, but that was only possible if interrupt entropy_nrf5_get_entropy_isr was triggered (I think), which didn't happen during the fuzzing session for some reason.

After I patched rng_pool_get to return 1, the coded proceeded a lot further since randomness is required to initialise many parts of the BT stack.

I wonder if there is a way to print the arm stack trace via hoedur (besides dump::stack which only prints raw addresses) or better expose the control flow via qemu? Currently I am printing all the symbols belonging to a basic block, but it's a bit difficult to pinpoint exactly when the qemu execution enters and leaves a function. Perhaps there is a way to hook all calls of all function (e.g., bl instruction)?.

• Trace of input generated during fuzzing session using interrupt mode fuzzed: log-fuzzed-isr.txt
• After patching rng_pool_get: log-test.txt
• Elf, bin and hordur config files: zephyr.tar.gz

[Matheus-Garbelini]

Thanks @SWW13 I was able to reach my main routine after some specific patches to the image. The tracing via pritnk and hooks helped to analyse where the emulation was stuck. Other than that hoedur can definitely figure it out many mmio access that would otherwise require manual reverse engineering.