radareorg / radare2

UNIX-like reverse engineering framework and command-line toolset
https://www.radare.org/
GNU Lesser General Public License v3.0
20.6k stars 2.99k forks source link

PE/x64 strange determine call convention #14737

Open s0i37 opened 5 years ago

s0i37 commented 5 years ago

Work environment

Questions Answers
OS/arch/bits (mandatory) Kali rolling x64
File format of the file you reverse (mandatory) PE (msvc VS2015 build)
Architecture/bits of the file (mandatory) x86/64
r2 -v full output, not truncated (mandatory) radare2 3.7.0-git 22725 @ linux-x86-64 git.3.6.0-265-g9677804b8

commit: 9677804b8e4f187656279a109d937c583ca0fb0a build: 2019-07-29__12:31:46

Expected behavior

I expected something like ELF/x64: temp1 there is r2 defines call convention well: arg1 rdi, arg2 rsi, arg3 rdx, arg4 rcx

Actual behavior

But for PE/x64 it is looks a bit strange: temp2 It seems to me that it could be like a: temp3 Also I met wrong arguments determine with dhcpssvc.dll: temp4

I couldn't reproduce this in my sample so I provide this binary. dhcpssvc.zip

Steps to reproduce the behavior

#include <stdio.h>
#include <stdlib.h>

int test(int a, int b, int c, int d)
{
    return a+b+c+d;
}

int main(void)
{
    printf("%d\n", test(1,2,3,4));
    return 0;
}

open VS2015 x64 console cl test.c

For dhcpssvc.dll:

idpd
idp
aaa
s pdb.UncodeOption
V
pelijah commented 5 years ago

Default cc for x86-64 binaries is amd64 (https://github.com/radare/radare2/blob/master/libr/anal/d/cc-x86-64.sdb.txt#L1). You can call afc=ms to change it to MS fastcall.

s0i37 commented 5 years ago

In general, If I open PE x86-64 binary, r2 determine call convention right.

s0i37 commented 5 years ago

I need to afc=ms before analisys aaa - it works. But how about changing call convention after analisys? I try afc ms but it doesn't change arguments.

thestr4ng3r commented 5 years ago

You have to re-analyze the function in any case, for example af for a single function.