NationalSecurityAgency / ghidra

Ghidra is a software reverse engineering (SRE) framework
https://www.nsa.gov/ghidra
Apache License 2.0
51.95k stars 5.9k forks source link

Wrong functions stack parameters offset for Dreamcast SH4 #4083

Open evpobr opened 2 years ago

evpobr commented 2 years ago

Describe the bug

There is no __cdecl calling convention for Dreamcast SH4.

To Reproduce

  1. Right click on function
  2. Select "Edit Function Signature"
  3. Click "calling convention" drop down list
  4. There are "unknown", "default", "stdcall" and "thiscall" choices, no "__cdecl"

Expected behavior

"calling convention" drop down list should contain "__cdecl".

Screenshots

Ghidra:

__cdecl_ghidra

IDA Pro:

__cdecl ida_pro

Attachments If applicable, please attach any files that caused problems or log files generated by the software.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

pjsoberoi commented 2 years ago

Should be simple enough to add another calling convention to the the SuperH4 cspec: https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Processors/SuperH4/data/languages/SuperH4_le.cspec and https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Processors/SuperH4/data/languages/SuperH4_be.cspec

Do you have a link to the Dreamcast cdecl calling convention?

evpobr commented 2 years ago

Hi @pjsoberoi . No, I don't have a specification, but I found someone who can help:

https://www.angelcode.com/angelscript/credits.html

  • Fredrik Ehnbom, a.k.a quarnster - alignment fixes, calling conventions for SH4 (Dreamcast) and ARM processors (WinCE, IPhone, Android), JIT compiler plug-in interface

@quarnster, could you help us, please?

evpobr commented 2 years ago

https://llvm-gcc-renesas.com/manuals/SH-ABI-Specification.html

pjsoberoi commented 2 years ago

Thanks for the link. It looks like gcc supports two calling conventions for SH4: a default calling convention and a Hitachi (-mhitachi) convention. I see information regarding how the arguments 4+ are pushed on the stack, but I'm not seeing anything regarding who cleans up the stack. stdcall the callee cleans up the stack, cdecl, the caller cleans up the stack.

Sorry I'm a bit stumped here.

evpobr commented 2 years ago

Found useful repo: https://github.com/Kochise/dreamcast-docs

https://github.com/Kochise/dreamcast-docs/blob/master/SDK/DOCS/Doc/PDFs/GNU_Emb.pdf

evpobr commented 2 years ago

But I do not know if it will be useful. I suspect that a MS compiler was used for my file:

https://segaretro.org/Windows_CE

pjsoberoi commented 2 years ago

In the IDA Pro example you have, the function is taking two args which will be passed via register. So cdecl\stdcall doesn't matter there. Do you have an example in IDA of a function taking more than 4 args? Do you have a way to compile a test binary with the Windows CE compiler?

evpobr commented 2 years ago

Do you have an example in IDA of a function taking more than 4 args?

``` .text:001AA170 ; =============== S U B R O U T I N E ======================================= .text:001AA170 .text:001AA170 .text:001AA170 void __cdecl DDBlit(struct IDirectDrawSurface4 *, struct tagRECT const &, struct IDirectDrawSurface4 *, struct tagRECT const &, unsigned long): .text:001AA170 ; CODE XREF: mouseManager::Update(bool)+388↑p .text:001AA170 ; mouseManager::Update(bool)+490↑p ... .text:001AA170 .text:001AA170 var_40 = -h'40 .text:001AA170 var_3C = -h'3C .text:001AA170 var_38 = -h'38 .text:001AA170 var_34 = -h'34 .text:001AA170 var_30 = -h'30 .text:001AA170 var_2C = -h'2C .text:001AA170 var_28 = -h'28 .text:001AA170 var_24 = -h'24 .text:001AA170 var_20 = -h'20 .text:001AA170 var_1C = -h'1C .text:001AA170 var_18 = -h'18 .text:001AA170 arg_10 = h'10 .text:001AA170 .text:001AA170 mov.l r8, @-r15 .text:001AA172 mov.l r9, @-r15 .text:001AA174 mov.l r10, @-r15 .text:001AA176 mov.l r11, @-r15 .text:001AA178 sts.l pr, @-r15 .text:001AA17A mov r4, r8 .text:001AA17C mov r5, r11 .text:001AA17E mov r6, r9 .text:001AA180 mov.l @(h'14+arg_10,r15), r10 .text:001AA182 add #-h'3C, r15 .text:001AA184 mov.l @r11, r5 .text:001AA186 mov.l @(4,r11), r4 .text:001AA188 mov.l r5, @(h'50+var_24,r15) .text:001AA18A mov.l r4, @(h'50+var_20,r15) .text:001AA18C mov.l @(h'C,r11), r5 .text:001AA18E mov.l @(8,r11), r4 .text:001AA190 mov.l r4, @(h'50+var_1C,r15) .text:001AA192 mov.l r5, @(h'50+var_18,r15) .text:001AA194 mov.l @r7, r4 .text:001AA196 mov.l @(4,r7), r5 .text:001AA198 mov.l r4, @(h'50+var_34,r15) .text:001AA19A mov.l r5, @(h'50+var_30,r15) .text:001AA19C mov.l @(8,r7), r4 .text:001AA19E mov.l @(h'C,r7), r5 .text:001AA1A0 mov.l r4, @(h'50+var_2C,r15) .text:001AA1A2 mov.l r5, @(h'50+var_28,r15) .text:001AA1A4 mov.l #IsRectEmpty, r0 .text:001AA1A6 mov #h'1C, r4 .text:001AA1A8 jsr @r0 ; IsRectEmpty .text:001AA1AA add r15, r4 .text:001AA1AC tst r0, r0 .text:001AA1AE bf loc_1AA2A2 .text:001AA1B0 mov.l #IDirectDrawSurface4 * lpFrontBuffer, r4 ; end of block .text:001AA1B2 mov.l @r4, r4 .text:001AA1B4 cmp/eq r4, r8 ; end of block .text:001AA1B6 bf loc_1AA1F2 .text:001AA1B8 mov.l #combatManager * gpCombatManager, r1 .text:001AA1BA mov.l @r1, r1 .text:001AA1BC mov.l @(h'34,r1), r0 .text:001AA1BE cmp/eq #1, r0 .text:001AA1C0 bf loc_1AA1EA .text:001AA1C2 mov.l #IDirectDrawSurface4 * MouseWorkSurface, r1 .text:001AA1C4 mov.l @r1, r1 .text:001AA1C6 cmp/eq r1, r9 .text:001AA1C8 bt loc_1AA2A2 .text:001AA1CA mov.l #IDirectDrawSurface4 * MouseSurface, r3 .text:001AA1CC mov.l @r3, r3 .text:001AA1CE cmp/eq r3, r9 .text:001AA1D0 bt loc_1AA2A2 .text:001AA1D2 mov.l #IDirectDrawSurface4 * SaveMouseSurface, r2 .text:001AA1D4 mov.l @r2, r2 .text:001AA1D6 cmp/eq r2, r9 .text:001AA1D8 bt loc_1AA2A2 .text:001AA1DA mov r11, r4 ; end of block .text:001AA1DA ; end of block .text:001AA1DC mov.l #int lastdx, r2 .text:001AA1DE mov.l #int lastdy, r3 .text:001AA1E0 mov.l @r3, r6 .text:001AA1E2 bsr DDAppBlitX(tagRECT const &,int,int) .text:001AA1E4 mov.l @r2, r5 .text:001AA1E6 bra loc_1AA1EE ; end of block .text:001AA1E6 ; end of block .text:001AA1E8 nop .text:001AA1EA ; --------------------------------------------------------------------------- .text:001AA1EA .text:001AA1EA loc_1AA1EA: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+50↑j .text:001AA1EA bsr DDAppBlit(tagRECT const &) .text:001AA1EC mov r11, r4 .text:001AA1EE .text:001AA1EE loc_1AA1EE: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+76↑j .text:001AA1EE bra loc_1AA2A2 ; end of block .text:001AA1EE ; end of block .text:001AA1F0 nop .text:001AA1F2 ; --------------------------------------------------------------------------- .text:001AA1F2 .text:001AA1F2 loc_1AA1F2: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+46↑j .text:001AA1F2 cmp/eq r4, r9 ; end of block .text:001AA1F2 ; end of block .text:001AA1F4 bf loc_1AA24C .text:001AA1F6 mov #h'2C, r5 ; ',' .text:001AA1F8 mov.l @(h'50+var_2C,r15), r6 .text:001AA1FA add r15, r5 .text:001AA1FC mov.l @(h'50+var_28,r15), r7 .text:001AA1FE mov r8, r4 .text:001AA200 mov.l r6, @(h'50+var_40,r15) .text:001AA202 mov.l r7, @(h'50+var_3C,r15) .text:001AA204 mov.l @(h'50+var_34,r15), r6 .text:001AA206 mov.l @(h'50+var_30,r15), r7 .text:001AA208 bsr sub_1AA16C .text:001AA20A mov.l r10, @(h'50+var_38,r15) .text:001AA20C bra loc_1AA2A2 .text:001AA20E nop .text:001AA20E ; --------------------------------------------------------------------------- .text:001AA210 word_1AA210: .data.w h'2D7 ; DATA XREF: DDRestoreFrontBuffer+24↑r .text:001AA212 word_1AA212: .data.w h'2DB ; DATA XREF: DDRestoreFrontBuffer+4C↑r .text:001AA214 word_1AA214: .data.w h'2DF ; DATA XREF: DDRestoreFrontBuffer+68↑r .text:001AA216 .align 4 .text:001AA218 off_1AA218: .data.l IsRectEmpty ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+34↑r .text:001AA21C off_1AA21C: .data.l short SCREEN_WIDTH .text:001AA21C ; DATA XREF: DDRestoreFrontBuffer:loc_1AA10E↑r .text:001AA220 off_1AA220: .data.l short SCREEN_HEIGHT .text:001AA220 ; DATA XREF: DDRestoreFrontBuffer+2C↑r .text:001AA224 off_1AA224: .data.l void * hwndApp ; DATA XREF: DDRestoreFrontBuffer+12↑r .text:001AA228 off_1AA228: .data.l `string' ; DATA XREF: DDRestoreFrontBuffer+22↑r .text:001AA228 ; DDRestoreFrontBuffer+4A↑r ... .text:001AA228 ; "E:\\gamedcs\\wingraph.cpp" .text:001AA22C off_1AA22C: .data.l combatManager * gpCombatManager .text:001AA22C ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+48↑r .text:001AA230 off_1AA230: .data.l IDirectDraw4 * lpDD .text:001AA230 ; DATA XREF: DDRestoreFrontBuffer+C↑r .text:001AA234 off_1AA234: .data.l IDirectDrawSurface4 * lpFrontBuffer .text:001AA234 ; DATA XREF: DDRestoreFrontBuffer:loc_1AA136↑r .text:001AA234 ; DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+40↑r .text:001AA238 off_1AA238: .data.l IDirectDrawSurface4 * MouseSurface .text:001AA238 ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+5A↑r .text:001AA23C off_1AA23C: .data.l IDirectDrawSurface4 * SaveMouseSurface .text:001AA23C ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+62↑r .text:001AA240 off_1AA240: .data.l IDirectDrawSurface4 * MouseWorkSurface .text:001AA240 ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+52↑r .text:001AA244 off_1AA244: .data.l int lastdx ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+6C↑r .text:001AA248 off_1AA248: .data.l int lastdy ; DATA XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+6E↑r .text:001AA24C ; --------------------------------------------------------------------------- .text:001AA24C .text:001AA24C loc_1AA24C: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+84↑j .text:001AA24C mov.l #GameTime::Get(void), r0 ; end of block .text:001AA24C ; end of block .text:001AA24C ; end of block .text:001AA24C ; end of block .text:001AA24C ; end of block .text:001AA24E jsr @r0 ; ?Get@GameTime@@YAKXZ .text:001AA250 nop .text:001AA252 .text:001AA252 loc_1AA252: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+130↓j .text:001AA252 mov #0, r3 .text:001AA254 mov.l r10, @(h'50+var_40,r15) .text:001AA256 mov #h'2C, r5 ; ',' .text:001AA258 mov.l r3, @(h'50+var_3C,r15) .text:001AA25A mov #h'1C, r7 .text:001AA25C mov.l @r8, r1 .text:001AA25E add r15, r7 .text:001AA260 mov.l @(h'14,r1), r1 .text:001AA262 mov r8, r4 .text:001AA264 mov r9, r6 .text:001AA266 jsr @r1 .text:001AA268 add r15, r5 .text:001AA26A mov r0, r11 .text:001AA26C mov.l #h'887601C2, r2 .text:001AA26E cmp/eq r2, r11 .text:001AA270 bf loc_1AA28E .text:001AA272 bsr DDRestoreSurfaces(void) .text:001AA274 nop .text:001AA276 tst r0, r0 .text:001AA278 bt/s loc_1AA284 .text:001AA27A mov r0, r4 .text:001AA27C mov.l #`string', r5 ; "E:\\gamedcs\\wingraph.cpp" .text:001AA27E mov.w #h'3DF, r6 .text:001AA280 bsr DDSD .text:001AA282 nop .text:001AA284 .text:001AA284 loc_1AA284: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+108↑j .text:001AA284 tst r11, r11 ; end of block .text:001AA286 mov #h'FFFFFFFF, r4 .text:001AA288 negc r4, r4 .text:001AA28A bra loc_1AA29E ; end of block .text:001AA28A ; end of block .text:001AA28A ; end of block .text:001AA28C nop .text:001AA28E ; --------------------------------------------------------------------------- .text:001AA28E .text:001AA28E loc_1AA28E: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+100↑j .text:001AA28E mov.l #h'887601AE, r1 .text:001AA290 cmp/eq r1, r11 .text:001AA292 bt loc_1AA2A2 .text:001AA294 tst r11, r11 ; end of block .text:001AA294 ; end of block .text:001AA296 mov #h'FFFFFFFF, r4 .text:001AA298 negc r4, r4 .text:001AA29A tst r4, r4 .text:001AA29C bf loc_1AA2A2 .text:001AA29E .text:001AA29E loc_1AA29E: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+11A↑j .text:001AA29E tst r4, r4 ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA29E ; end of block .text:001AA2A0 bf loc_1AA252 .text:001AA2A2 .text:001AA2A2 loc_1AA2A2: ; CODE XREF: DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+3E↑j .text:001AA2A2 ; DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong)+58↑j ... .text:001AA2A2 add #h'3C, r15 ; '<' .text:001AA2A4 lds.l @r15+, pr .text:001AA2A6 mov.l @r15+, r11 .text:001AA2A8 mov.l @r15+, r10 .text:001AA2AA mov.l @r15+, r9 .text:001AA2AC rts .text:001AA2AE mov.l @r15+, r8 .text:001AA2AE ; End of function DDBlit(IDirectDrawSurface4 *,tagRECT const &,IDirectDrawSurface4 *,tagRECT const &,ulong) ```

Do you have a way to compile a test binary with the Windows CE compiler?

I've never done that.

evpobr commented 2 years ago

@pjsoberoi , after some researching i guess i found the answer. The problem is not missing cdecl.

MS compiler puts first 4 arguments in registers R4-R7, and puts other arguments on stack and Ghidra knows it. But Ghidra starts 5th argument with offset 0 (stack[0x0]), but MS compiler reserves extra space (sum of R4-R7 registers), so it should be 16 (stack[0x10]).

https://docs.microsoft.com/fr-fr/previous-versions/visualstudio/visual-studio-2008/ms253572(v=vs.90)?redirectedfrom=MSDN

fenugrec commented 2 years ago

For the record, binaries compiled with Hitachi / Renesas C/C++ compiler follow the calling convention detailed in the document # R20UT0704EJ0102 ( https://www.renesas.com/us/en/document/mat/superh-cc-compiler-package-v904-users-manual?language=en&r=1054751 ) ;

Table 9.6 " General Rules on Parameter Area Allocation" , p.319

R4 to R7 char, unsigned char, bool, short, unsigned short, int, unsigned int, long, unsigned long, float (when CPU is other than SH-2E, SH2A- FPU, SH-4, or SH-4A), pointer, pointer to a data member, and reference

Parameters Allocated to a Stack (1) Parameters whose types are other than target types for register passing (2) Parameters of a function which has been declared by a prototype declaration to have variable-number parameters*3 (3) When other parameters are already allocated to R4 to R7. (4) When other parameters are already allocated to FR4 (DR4) to FR11 (DR10). (5) long long type and unsigned long long type parameters

(6) _ fixed type, long fixed type, accum type, and long _accum type parameters

evpobr commented 2 years ago

In general, I now manually set the addresses of the parameters that are passed in the stack with an offset of 0x10 and this solves the problem. But I would like an official fix. I suspect this affects the analysis of the code as a whole.