Why don't you maintain this project any more?

[Bl1nnnk]

Such a powerful project.

And I have found and fixed several bugs, do you mind that I maintain it in my forked repository?

[eqv]

This repo mostly serves as a reference for our papers. We are actively working on this project internally, the most recent release of the backend is found here https://github.com/RUB-SysSec/redqueen and of the frontend here https://github.com/RUB-SysSec/grimoire. We will soon release a completely new, significantly improved version. Also, in the next few weeks, others will publish a branch of kAFL/redqueen/grimoire with company backing the development. You might want to wait a few days and join efforts. Otherwise you are obviously free to maintain a version yourself.

[Bl1nnnk]

I see. Thanks, looking forward to the next version:)

[eqv]

For the record: Bl1nnnk's patches are available here: https://github.com/Bl1nnnk/kAFL And there is a more up to date, (and for now) actively maintained version published by intel here: https://github.com/IntelLabs/kAFL/

[hMihaiDavid]

Does the intel version contain Bl1nnnk's patches? thanks.

[eqv]

I don't think it does. But they are based on the Redqueen back-end which already included a significant number of improvements, and there is a chance this already fixed some of the bugs. I also think, they will be merging the remaining patches soon.

[il-steffen]

@Bl1nnnk Could you give some detail on the fixes in decoder.c/disassember.c. How can I confirm the bug? The Redqueen code is just different enough to make this non-trivial to apply, and I'm also not sure if every line in your patch is intended as is: https://github.com/Bl1nnnk/kAFL/blob/master/QEMU-PT/pt/disassembler.c#L499 ;-)

[Bl1nnnk]

You know that pt package decoding have to be combined with binary code. The kAFL read all the guest binary code during initialization: https://github.com/RUB-SysSec/kAFL/blob/master/QEMU-PT/pt.c#L204. But some executable pages maybe page-out even in kernel mode.

So I have qemu-pt fetch the guest binary code after execution in that page:

1. Read and parse the guest code no more than one page. get_cofi_ptr is used to deal with the situation that the block/instruction is across pages.
2. If the block/instruction is across pages when it being parsed, the second page must have been excuted. Then the second page could be read and parsed safely.

L490/L499/L508 is useless, I forgot to remove them :)

[e13fter]

Hi, @il-steffen! Have you included @Bl1nnnk's patches to the IntelLabs version of kAFL? I'm trying to fuzz linux kafl_vuln_test module and at last it seems to work but sees even the first crash at "KERNELAFL" input as not new, therefore it skips it. I guess the bitmap is wrong and wonder if it can be related to things @Bl1nnnk describes. Thank you

[il-steffen]

@e13fter I reviewed the patch and applied best I could. See https://github.com/IntelLabs/kAFL/issues/9. There is still a fundamental issue that memory may be paged out during trace decoding, likely also some other bugs in decoder.c. About 1% of the executions can have a couple wrong edges in the bitmap. I added a -funky mode to work around non-determinism issues like this. I can also confirm this is fixed in the Nyx/libxdc release but is not public yet and not easy to backport to kAFL.

I don't think any of this relates to the problem you mention, since crashes are matched against a separate bitmap. Are you getting any feedback at all? If the bitmap is empty, a crashing input may be discarded as "not new".

I suggest to start from scratch using one of the recent confirmed tutorials https://github.com/IntelLabs/kAFL/issues/27#issuecomment-757012550 or https://github.com/IntelLabs/kAFL/pull/31.

[e13fter]

Seems like there are really empty bitmaps. I slighly modified agent for it to output payload through hprintf and see that there are no "fixed" hit values in payload. Even values which are to increase coverage, as I know from the source code, are treated as all others. When I provide an "almost" crashing input, like "KERNELoFL" after some random mutations it leads to the crash, of course, and the fuzzer prints this:

[HPRINTF]   00000000: 4b 45 52 4e 45 4c 5b 46 4c 0a d3 d3 20 e4 e4 fa "KERNEL[FL... ..."
[HPRINTF]   00000000: 4b 45 52 4e 45 4c 5c 46 4c 0a d3 d3 20 e4 e4 fa "KERNEL\FL... ..."
[HPRINTF]   00000000: 4b 45 52 4e 45 4c 42 46 4c 0a d3 d3 20 e4 e4 fa "KERNELBFL... ..."
[HPRINTF]   00000000: 4b 45 52 4e 45 4c 5d 46 4c 0a d3 d3 20 e4 e4 fa "KERNEL]FL... ..."
[HPRINTF]   00000000: 4b 45 52 4e 45 4c 41 46 4c 0a d3 d3 20 e4 e4 fa "KERNELAFL... ..."
Crashing input found (crash), but not new (discarding) 0

If I provide second test case, the fuzzer takes it, runs and considers it to produce no new coverage and finally goes on with dumb mutations without any feedback.

[HPRINTF]   00000000: 4b 45 52 4e 45 4c ff ff ff 7f d3 d3 20 e4 e4 fa "KERNEL...... ..."
Importing payload from /home/user/Documents/fuzz/tools/kafl/fuzz/imports/1
[HPRINTF]   00000000: 53 45 52 31 32 33 0a ff ff 7f d3 d3 20 e4 e4 fa "SER123...... ..."
Imported payload produced no new coverage, skipping..
[HPRINTF]   00000000: 00 20 52 31 32 33 0a ff ff 7f d3 d3 20 e4 e4 fa ". R123...... ..."

Looking through python part of the fuzzer and its native bitmap library I think there is no mistakes in it and suspect the tracing PT part of KVM and QEMU. How could I debug it? I tried to enable PT tracing via the QMP monitor, but it failed.

(qemu) pt enable_all 
CPU 0: failed...

Scared to dig into Intel PT insides and hope that the problem can be solved without it.

By the way, I removed the vm_ram option from the command line, because with it qemu couldn't find the right snapshot when launching kafl_info.py or kafl_fuzz.py. Also I adjusted the regular expressions in pt/asm_decoder.c to fix the problem from this issue https://github.com/IntelLabs/kAFL/issues/18. As you told there it is the redqueen part, so I guess it doesn't affect my kernel fuzzing.

My CPU is Intel Core i7-8550U (Kaby Lake R).

Thank you

[il-steffen]

This looks like an issue with the overall setup. Try to follow the mentioned guides exactly, and if that doesn't work try the Zephyr/UEFI examples. I think newer Linux guests are making problems and you probably have to disable spectre mitigations or kpti or something like that. If the problem persists, please post a new issue in the appropriate board and give some background on your installation/steps.

[e13fter]

Of course I disabled spectre mitigations, kpti, KASLR - all things I found in other issues. Also I've tried another setup. First, I tried another desktop motherboard: Gigabyte Q370-M. IntelLabs' KAFL worked with the same problems I described. But due to the fact that the BIOS on that motherboard supports legacy bootloaders, I could install old Ubuntu 16.04 as the host operating system and installed RUB-SysSec's KAFL there. And it worked right, well and as expected! Second, I tried all these actions on Debian 10.2, as it is in your kernel config here https://github.com/IntelLabs/kAFL/issues/18#issuecomment-706700129. Same result. So, I don't know what else to try. I found some inaccuracies in the code and want to open another issue.

[il-steffen]

If the old kAFL is working, can you try some intermediate versions of KVM-PT and Qemu-PT to isolate the problem? The 4.x kernel module should actually be compatible, and all the Qemu changes are stored as separate patches on top of the RUB-SysSec baseline: https://github.com/IntelLabs/kAFL/tree/master/patches/qemu/v5.0.0

If you go back in history there is also a v3.1 qemu that should be identical to the SysSec version. The problems you observed in capstone/decoder.c may be related to relatively recent patch 0012 or 0014.

[e13fter]

I tried old kAFL kvm part from RUB SysSec and v3.1 qemu from IntelLabs. The fuzzer hangs after Stage 1 handshake done [INIT]. When launching v5.0 qemu from IntelLabs on the old kvm, kernel crashes. So, seems not quite compatible.

I moved this discussion to the separate issue https://github.com/IntelLabs/kAFL/issues/32.