Segmentation fault when decompiling

[TambourineReindeer]

This has happened with every file I've tried to decompile, mostly Windows exes, but a few elf files as well. At the decompiling stage, Boomerang will get about halfway and immediately crash around the same point every time. The only way I've been able to circumvent this is to check "Don't decode child processes", but that greatly reduces the code output to a point that's not useful. "Enable data-flow analysis" does not make a change in whether or not it crashes.

I compiled and ran Boomerang on Linux Mint 17.2 Cinnamon 64-bit.

Here are some executables I've tried: http://advsys.net/ken/voxlap/pnd3d.htm https://drive.google.com/file/d/0B0Tw1fnDScRsN192NzRqYk8yckE/view http://www.classicdosgames.com/game/Inner_Worlds.html

I may later try to debug in gdb and provide more information if possible. Console output (sorry, won't let me attach it):

---

./boomerang Not adding 0 sized section "" Signatures directory does not exist. Valid but undecoded instruction at "63a1f" Valid but undecoded instruction at "63a4f" Valid but undecoded instruction at "676b5" Valid but undecoded instruction at "675d5" Valid but undecoded instruction at "2387" Valid but undecoded instruction at "25a3" ERROR: unknown instruction INTIB at 4351, ignoring. Valid but undecoded instruction at "4351" ERROR: unknown instruction INTIB at 435e, ignoring. Valid but undecoded instruction at "435e" Valid but undecoded instruction at "404b" Valid but undecoded instruction at "63a1f" Valid but undecoded instruction at "63a4f" Valid but undecoded instruction at "67735" Valid but undecoded instruction at "678b5" Valid but undecoded instruction at "675f5" Valid but undecoded instruction at "67775" Valid but undecoded instruction at "3e82" Valid but undecoded instruction at "5e83" Valid but undecoded instruction at "61d2" Valid but undecoded instruction at "3ca9b" Valid but undecoded instruction at "468b" Valid but undecoded instruction at "ab17" Valid but undecoded instruction at "4cddb" Valid but undecoded instruction at "90cf" ERROR: unknown instruction INTIB at 3fc1, ignoring. Valid but undecoded instruction at "3fc1" ERROR: unknown instruction INTIB at 3fce, ignoring. Valid but undecoded instruction at "3fce" Valid but undecoded instruction at "65d77" Valid but undecoded instruction at "663b3" Valid but undecoded instruction at "65fd7" Valid but undecoded instruction at "5e127" Valid but undecoded instruction at "677b5" Valid but undecoded instruction at "3583" Valid but undecoded instruction at "6053" Valid but undecoded instruction at "3ea7" ERROR: unknown instruction INTIB at 43a5, ignoring. Valid but undecoded instruction at "43a5" ERROR: unknown instruction INTIB at 43c9, ignoring. Valid but undecoded instruction at "43c9" Valid but undecoded instruction at "11b9f" Valid but undecoded instruction at "6c17" Valid but undecoded instruction at "45ff" Valid but undecoded instruction at "13c87" Valid but undecoded instruction at "646db" Valid but undecoded instruction at "cd47" Valid but undecoded instruction at "454b" ERROR: unknown instruction INTIB at 4551, ignoring. Valid but undecoded instruction at "4551" ERROR: unknown instruction INTIB at 4565, ignoring. Valid but undecoded instruction at "4565" Valid but undecoded instruction at "4cd8b" Valid but undecoded instruction at "846b" Valid but undecoded instruction at "3dea" Valid but undecoded instruction at "63c3b" Valid but undecoded instruction at "676f5" Valid but undecoded instruction at "3113" Valid but undecoded instruction at "32b7" Valid but undecoded instruction at "3457" Valid but undecoded instruction at "17ef5" Valid but undecoded instruction at "39a3" Valid but undecoded instruction at "196f7" Valid but undecoded instruction at "196be" Valid but undecoded instruction at "1974a" Valid but undecoded instruction at "196da" Valid but undecoded instruction at "10ddf" Valid but undecoded instruction at "f5df" Valid but undecoded instruction at "157cb" Valid but undecoded instruction at "64985" Valid but undecoded instruction at "ba73" Valid but undecoded instruction at "67635" Valid but undecoded instruction at "62deb" Valid but undecoded instruction at "63037" Valid but undecoded instruction at "67615" Valid but undecoded instruction at "54913" Valid but undecoded instruction at "1107b" Valid but undecoded instruction at "5bdb7" Valid but undecoded instruction at "76d3" Valid but undecoded instruction at "7737" Valid but undecoded instruction at "409b" Valid but undecoded instruction at "677d5" Valid but undecoded instruction at "1a407" Valid but undecoded instruction at "1a417" Valid but undecoded instruction at "67bdf" Valid but undecoded instruction at "67675" Valid but undecoded instruction at "5f4f6" Valid but undecoded instruction at "4ce0f" Valid but undecoded instruction at "17acb" Valid but undecoded instruction at "11a0f" Valid but undecoded instruction at "67715" Valid but undecoded instruction at "5c3fb" Valid but undecoded instruction at "749b" Valid but undecoded instruction at "1355f" Valid but undecoded instruction at "48a4" Valid but undecoded instruction at "4b57" Valid but undecoded instruction at "1be9f" Valid but undecoded instruction at "18467" Valid but undecoded instruction at "3984" Valid but undecoded instruction at "38f7" Valid but undecoded instruction at "392d" Valid but undecoded instruction at "19037" Valid but undecoded instruction at "19627" Valid but undecoded instruction at "1863b" Valid but undecoded instruction at "5feab" Valid but undecoded instruction at "5fc07" Valid but undecoded instruction at "5facf" Valid but undecoded instruction at "5fbaf" Valid but undecoded instruction at "5fbb2" Valid but undecoded instruction at "5fbce" Valid but undecoded instruction at "5fb6e" Valid but undecoded instruction at "5d4b3" Valid but undecoded instruction at "5d88f" Valid but undecoded instruction at "60220" Valid but undecoded instruction at "5fa93" Valid but undecoded instruction at "5fa4a" Valid but undecoded instruction at "6016c" Valid but undecoded instruction at "5eb78" Valid but undecoded instruction at "5f0b4" Valid but undecoded instruction at "676d5" Valid but undecoded instruction at "648af" Valid but undecoded instruction at "17d9a" Valid but undecoded instruction at "67655" Valid but undecoded instruction at "631a7" Valid but undecoded instruction at "162cb" Valid but undecoded instruction at "1c093" Valid but undecoded instruction at "19237" Valid but undecoded instruction at "185f7" Valid but undecoded instruction at "5fa6c" Valid but undecoded instruction at "67895" Valid but undecoded instruction at "5c7f3" Valid but undecoded instruction at "64b1b" Valid but undecoded instruction at "ffdf" Valid but undecoded instruction at "1417f" Valid but undecoded instruction at "14347" Valid but undecoded instruction at "1407f" Valid but undecoded instruction at "19ea3" Valid but undecoded instruction at "17fb7" Valid but undecoded instruction at "191c7" main function not found main function not found main function not found Error: no entry for INT.Ib' in RTL dictionary Error: no entry forINT.Ib' in RTL dictionary Error: no entry for INT.Ib' in RTL dictionary Error: no entry forINT.Ib' in RTL dictionary Error: no entry for INT.Ib' in RTL dictionary Error: no entry forINT.Ib' in RTL dictionary Error: no entry for INT.Ib' in RTL dictionary Error: no entry forINT.Ib' in RTL dictionary considering _start considering proc1 considering proc8 considering proc15 considering proc4 decompiling proc4 considering proc7 considering proc14 decompiling proc14 t decompiling proc7 rprrpttttr decompiling proc15 prtttr decompiling proc8 prprrrrprpttttttttttr decompiling proc1 pr considering proc2 decompiling proc2 considering proc3 considering proc9 considering proc16 considering proc86 considering proc191 considering proc315 decompiling proc315 considering proc316 considering proc440 considering proc382 decompiling proc382 decompiling proc440 rtt considering proc441 considering proc521 considering proc561 decompiling proc561 decompiling proc521 p decompiling proc441 decompiling proc316 rtt decompiling proc191 rprt considering proc192 considering proc17 considering proc87 decompiling proc87 rtt considering proc6 decompiling proc6 prt decompiling proc17 t considering proc193 considering proc317 considering proc442 decompiling proc442 r considering proc363 decompiling proc363 rt decompiling proc317 prrprrrpttttttttr decompiling proc193 rpprrrprrrrprrpprValid but undecoded instruction at "65d77" Valid but undecoded instruction at "663b3" Valid but undecoded instruction at "65fd7" Valid but undecoded instruction at "67835" Valid but undecoded instruction at "677f5" Valid but undecoded instruction at "67815" reconsidering proc86 considering proc576 considering proc577 considering proc583 considering proc586 decompiling proc586 considering proc587 decompiling proc587 considering proc588 decompiling proc588 considering proc589 decompiling proc589 considering proc590 decompiling proc590 considering proc591 considering proc592 decompiling proc592 decompiling proc591 rt considering proc21 considering proc5 decompiling proc5 decompiling proc21 prrtttr decompiling proc583 prttr decompiling proc577 prtt considering proc582 decompiling proc582 rprtttp considering proc203 decompiling proc203 rrpttt considering proc585 decompiling proc585 rprttttr considering proc584 decompiling proc584 prprttttt decompiling proc576 rprprtttt considering proc578 decompiling proc578 rValid but undecoded instruction at "69fc3" Valid but undecoded instruction at "69e07" Valid but undecoded instruction at "6a51b" Valid but undecoded instruction at "6a5c7" considering proc579 decompiling proc579 considering proc580 considering proc593 considering proc594 decompiling proc594 considering proc595 considering proc596 decompiling proc596 prt considering proc606 decompiling proc606 r considering proc198 considering proc600 decompiling proc600 rp considering proc599 decompiling proc599 considering proc597 considering proc603 decompiling proc603 rttp considering proc598 decompiling proc598 rprtttt decompiling proc597 prtt considering proc601 decompiling proc601 rpttrValid but undecoded instruction at "68563" Valid but undecoded instruction at "6982b" Valid but undecoded instruction at "6659b" considering proc602 considering proc605 considering proc604 considering proc581 decompiling proc581 ttrprppprrpprrrrpprrrrrrpprrrrrrrpprrrrrrpprrrppprrValid but undecoded instruction at "69fc3" Valid but undecoded instruction at "69e07" Valid but undecoded instruction at "6a51b" Valid but undecoded instruction at "6a5c7" reconsidering proc593 rpprprpprrrrrpprrrrrpprrrrrrrpprrrrrrrpprrrppprrrrprprrprrrrrpprrrrrrrpprrrppprrrppppppttttttttttttttttttttttttttttttttrpprtttrtttttttpttttttttttttttttttttttttttttttttttttttttttttrrppprrtrrrrSegmentation fault

[nemerle]

Thank you for this - free time permitting I'll look into the first two of those. Sadly the third is a DOS executable, and boomerang does not handle those at all :disappointed:

Leaving this open - will update if anything changes

[TambourineReindeer]

Okay, there are a lot more, essentially all the files I've worked with, but I can't remember them all. >.< I didn't know Boomerang didn't handle DOS executables. I actually was able to extract an elf executable from it (in my attempts at reverse engineering, using photorec to scan), and that Boomerang was able to read, but it ran into the same problem as the others with crashing partway through.

Thank you very much though :)

[TambourineReindeer]

Here's the elf executable: https://www.mediafire.com/?49oajbhqf39nbpr Unfortunately the only thing that happens when I try to run it from the command line is a message to the console saying "Killed". I'm assuming since it's an old executable from the 1990s, it probably isn't runnable even if it is a Linux executable.

---

Pastes of GDB logs (one is the attempted decompile of the elf file I mentioned, the other is a dll file) http://pastebin.com/sarZKyhj http://pastebin.com/JgDWV3jb

[TambourineReindeer]

I haven't been able to look at this too much, but I skimmed through the GDB logs again and it appears the second one involves an assertion in proc.cpp failing (line 5627, or assert(pas.second.def()); ). The first one is that it seems to have a problem with the iterators within UserProc::recursionGroupAnalysis(), or UserProc::decompile() itself which is called inside recursionGroupAnalysis().

I hope this helps and I'm sorry the GDB logs are a mess to read.

And is the console supposed to output "p" "r" and "t" or is that a bug?

[nemerle]

The main problem seems to be with recursionGroupAnalysis and indirect control transfer processing.

At one point the dataflow and proc->cfg synchronization seems to be lost ( the size of dataflow->indices that should be exactly same as proc->getCFG()->getNumBBs() is no longer same ) - needs more investigation

[JustGregory-zz]

@TambourineReindeer : "I'm assuming since it's an old executable from the 1990s, it probably isn't runnable even if it is a Linux executable."

... Well, not necessarily un-runnable. If you can run VirtualBox or other virtualization software, and if you can find a "free dos" or MS-DOS alternative, you may be able to run it, ... I think.

[uxmal]

@TambourineReindeer: until Boomerang does support MS-DOS, may I humbly suggest trying your binary with the reko decompiler (https://github.com/uxmal/reko)?

[TambourineReindeer]

Thank you all of you. @uxmal I'll look at Reko soon :3 @JustGregory I can get the executable to run in DOSBox just fine (which has a debug mode as well but I had trouble with that) and I've used it in FreeDOS as well with some success (runnable in VirtualBox but no audio so I might have been missing something). I learned later that the DOS program uses a DOS Extender called MOSS DOS extender (http://www.cs.utah.edu/flux/moss/) (http://www.bttr-software.de/forum/forum_entry.php?id=5865) and is ELF-based, hence the ELF file inside it (which decompiles under most decompilers).

@nemerle Maybe if I recompile Boomerang with -g I can get better output to help diagnose the problem.

[JustGregory-zz]

Ah, okay, my mind must've skipped over that you said "elf-based" and failed to connect it to "Linux executable".

[ceeac]

I'm closing this issue since Boomerang does not crash for the ELF any more, neither when using develop nor when using the capstone branch. Note that many instructions are still not recognized, so the output will not be correct. Please open a new issue if Boomerang still crashes.