search

avast / retdec

RetDec is a retargetable machine-code decompiler based on LLVM.
https://retdec.com/
MIT License
8.03k stars 950 forks source link

Unhandled type conversion introduced by new demangler #605

Open leftmostcat opened 5 years ago

leftmostcat commented 5 years ago

Attempting to decompile the demo at https://www.gamepressure.com/download.asp?ID=7338 results in the following error:

convertValueToType(): unhandled type conversion
  call void @_free(i32* @317)
i32

After bisecting, ed685a6fde4a4ca096a3ef02c32edd9d432b4503 is indicated as the culprit commit.

s3rvac commented 5 years ago

Hi. I have tried decompiling the input binary file via retdec-decompiler.py lego_star_wars_pc_demo_final.exe with the debug build of the current master on Linux, but I am unable to reproduce the issue. I have also tried disabling the unpacking step, but the decompilation still succeeds.

Could you please provide the following information?

twitzelbos commented 5 years ago

I have a similar issue on decompiling a binary. Strangely enough, with the Release build of retdec I get a segmentation fault in LLVM (in the instruction combiner). I made a debug build of retdec, now I get this assert failure instead on the same target binary/command line. Will dig into this some more. I'm using msys2.

leftmostcat commented 4 years ago

My original issue description was unclear, sorry. I am installing the demo using the file above and decompiling the resulting LegoStarwars.exe via retdec-decompiler.py LegoStarwars.exe. I don't recall the commit I originally filed against, but it persists with latest master (b5f06019a8577cef131c5375a843b6f9974dad43). I am using Fedora Linux 31 at present. Full output as follows:


##### Checking if file is a Mach-O Universal static library...

##### Checking if file is an archive...
RUN: /home/sdeburca/.local/bin/retdec-ar-extractor /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe --arch-magic
Not an archive, going to the next step.

##### Gathering file information...
RUN: /home/sdeburca/.local/bin/retdec-fileinfo -c /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe.json --similarity /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe --no-hashes=all --crypto /home/sdeburca/.local/bin/../share/retdec/support/generic/yara_patterns/signsrch/signsrch.yara --crypto /home/sdeburca/.local/bin/../share/retdec/support/generic/yara_patterns/signsrch/signsrch.yarac --max-memory-half-ram
Input file               : /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe
File format              : PE
File class               : 32-bit
File type                : Executable file
Architecture             : x86
Endianness               : Little endian
Image base address       : 0x400000
Entry point address      : 0x615acb
Entry point offset       : 0x215acb
Entry point section name : .text
Entry point section index: 0
Bytes on entry point     : 6a606838f36c00e839410000bf940000008bc7e8adfeffff8965e88bf4893e56ff15a07062008b4e10890de8bc31018b4604
Detected tool            : Microsoft Linker (7.1) (linker), combined heuristic
Detected tool            : MSVC (7.1) Visual Studio .NET 2003 (compiler), combined heuristic
Detected tool            : Microsoft (linker), dos header style
Original language        : C++
Rich header offset       : 0x80
Rich header key          : 0xf74e0fd8
Rich header signature    : 0069081300000002000f0c0500000031005f088300000002005d081300000002005d0c0500000002001c23da
                           00000003001923fa00000004005d08830000000a005d0fc30000000300010000000000b3005f0fc300000001
                           005f08130000000c001220fc0000000600600fc30000007f001d227e00000001005f0c050000012800600c05
                           0000002c005e0bec00000001005a0c0500000001

##### Trying to unpack /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe into /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe-unpacked.tmp by using generic unpacker...
RUN: /home/sdeburca/.local/bin/retdec-unpacker /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe -o /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe-unpacked.tmp --max-memory-half-ram
No matching plugins found for 'Microsoft Linker 7.1'
No matching plugins found for 'MSVC 7.1'
No matching plugins found for 'Microsoft'
##### Unpacking by using generic unpacker: nothing to do
##### 'upx' not available: nothing to do

##### Decompiling /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe into /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe.bc...
RUN: /home/sdeburca/.local/bin/retdec-bin2llvmir -provider-init -decoder -verify -x87-fpu -main-detection -idioms-libgcc -inst-opt -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -x86-addr-spaces -register-localization -value-protect -instcombine -tbaa -basicaa -simplifycfg -early-cse -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -gvn -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -dse -bdce -adce -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instcombine -instcombine -tbaa -basicaa -simplifycfg -early-cse -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -gvn -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -dse -bdce -adce -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instcombine -inst-opt -simple-types -stack-ptr-op-remove -idioms -instcombine -inst-opt -idioms -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe.json -max-memory-half-ram -o /home/sdeburca/Tionscadail/lswdec/LegoStarwars.exe.bc
Running phase: Initialization ( 0.03s )
Running phase: LLVM ( 0.06s )
Running phase: Providers initialization ( 0.06s )
Running phase: Input binary to LLVM IR decoding ( 0.89s )
Running phase: LLVM ( 14.26s )
Running phase: x87 fpu register analysis ( 15.09s )
Running phase: Main function identification optimization ( 15.37s )
Running phase: Libgcc idioms optimization ( 15.40s )
Running phase: LLVM instruction optimization ( 15.40s )
Running phase: Conditional branch optimization ( 15.82s )
Running phase: Syscalls optimization ( 19.04s )
Running phase: Stack optimization ( 19.04s )
Running phase: Constants optimization ( 26.93s )
Running phase: Function parameters and returns optimization ( 57.53s )

convertValueToType(): unhandled type conversion
  call void @_free(i32* @317)
i32

Running phase: Register localization optimization ( 65.18s )
Running phase: LLVM instruction optimization ( 70.30s )
Running phase: Simple types recovery optimization ( 70.80s )
Running phase: Disassembly generation ( 76.88s )
Running phase: Assembly mapping instruction removal ( 83.86s )
Running phase: C++ class hierarchy optimization ( 87.75s )
Running phase: Selected functions optimization ( 88.16s )
Running phase: Unreachable functions optimization ( 88.16s )
Running phase: LLVM instruction optimization ( 88.42s )
Running phase: x86 address spaces optimization ( 88.87s )
Running phase: Make all registers local ( 89.08s )
Running phase: Value protection optimization ( 89.48s )
Running phase: LLVM ( 89.85s )
Error: Decompilation to LLVM IR failed```