amd64 shellcode crashes when called with a 16-byte aligned stack pointer

[kylewillmon]

From my testing, it appears that donut shellcode crashes if it gains execution with rsp % 16 == 0 instead of rsp % 16 == 8

Microsoft's x64 calling convention requires that the stack pointer must be 16-byte aligned before any call instruction. So code that is meant to be called expects that rsp % 16 == 8 when it gains executions (because the call pushes 8 bytes onto the stack). Some code takes advantage of this alignment when using XMM operations which only work with aligned data.

In my case, executing donut with a 16-byte aligned stack eventually causes a crash deep inside a LoadLibrary() call due to one of these XMM operations. However, different versions of Windows may crash at different locations.

Donut should handle both cases because there are many different ways to execute shellcode.

I have been testing this with a simple shellcode loader that I wrote for this testing. Here are the commands if you want to try it yourself with that loader.

To generate the shellcode:

donut\donut.exe -a2 -b1 -o test.bin -x2 -z2 hello-world.exe

Running with tail call works:

shellcode-rs.exe -m tail test.bin

Running with ret call crashes:

shellcode-rs.exe -m ret test.bin