NationalSecurityAgency / ghidra

Ghidra is a software reverse engineering (SRE) framework
https://www.nsa.gov/ghidra
Apache License 2.0
49.16k stars 5.66k forks source link

Ability to have more than `Byte.MAX_VALUE` calling conventions #5338

Open lab313ru opened 1 year ago

lab313ru commented 1 year ago

Subj. This can be used to generate custom calling conventions with different input and output destinations. For now you cannot apply more than 127 calling conventions using HighFunctionDBUtil.writeOverride, it prints

ERROR Unable to assign calling convention `__r39_r56` - allocation capacity exceeded (CallingConventionDBAdapterV0)  
ghidra1 commented 1 year ago

That seems rather excessive and is likely abusing the intention of reuseable "named" calling conventions. I assume this is being done to customize storage on a function-by-function basis or do you intended on reusing each named convention to multiple functions? Our intent is to add custom storage capability to Function Definitions in a manner similar to that of Function.

jibee commented 2 months ago

I am reaching a similar issue here reverse engineering hand-generated code; so without a consistent calling "convention".

The information from the "calling convention" I do not see in the custom storage is the list of affected (and symmetrically, unaffected) registers; from which the true output registers can be drawn (using the custom storage).

A large number of calling convention allows for a fine-grained analysis, without which I would be missing the complete effect of the function (returning a structure across multiple registers) or on the contrary been swamped with non-existant side-effects.

But in my case being able to set the killed by / unaffected by lists on a function by function basis would also solve my issue.

ghidra1 commented 2 months ago

For taylored situations as you describe (e.g., pure assembly code) we are evaluating an approach which would allow a function-specific tayloring of affected/unaffected registers, etc.. We are also evaluating similar storage-specific details for function definitions (i.e., data type). We prefer not to blow-out the number of defined calling conventions used at this point in time.