search

cmu-sei / pharos

Automated static analysis tools for binary programs
Other
1.57k stars 192 forks source link

Unknown message: error(system_error(initialSanityChecks)) error from ooprolog #261

Closed Shenmarukai closed 8 months ago

Shenmarukai commented 8 months ago

So essentially I'm trying to recover C++ class information from a 32 bit - PE-i386 format - executable compiled with MSVC. Im unsure of what version of MSVC was used to compile the executable, but I do have information on when it was compiled so the version of MSVC must be before this date -> Tue Dec 9 13:22:45 2003, so I think it must have been compiled with Visual C++ .NET 2003 / Visual C++ 7.1 or earlier:

F3.exe: file format pei-i386

executable 32 bit words

Time/Date Tue Dec 9 13:22:45 2003 Magic 010b (PE32) MajorLinkerVersion 7 MinorLinkerVersion 0 MajorOSystemVersion 4 MinorOSystemVersion 0 MajorImageVersion 0 MinorImageVersion 0 MajorSubsystemVersion 4 MinorSubsystemVersion 0 Win32Version 00000000 Subsystem 00000002 (Windows GUI)

I installed pharos via the docker image from:

I am running the docker image via:

The following is a summary of the outputs of each step. I can also provide the .exe file, .ser file, facts.pl file, results.pl file, and/or .log file if required.

Partitioning is successful:

OPTI[INFO ]: Analyzing executable: /data/F3.exe
OPTI[INFO ]: ROSE stock partitioning took 14.9721 seconds.
OPTI[INFO ]: Partitioned 731129 bytes, 244954 instructions, 56272 basic blocks, 65 data blocks and 3480 functions.
OPTI[INFO ]: Function partitioning took 677.933 seconds.
OPTI[INFO ]: Writing serialized data to "/data/F3.ser".
OPTI[INFO ]: Writing serialized data took 67.9488 seconds.
OPTI[INFO ]: Partitioned 2384469 bytes, 731889 instructions, 171589 basic blocks, 10343 data blocks and 17533 functions.

Analysis I think is successful:

OPTI[INFO ]: Analyzing executable: /data/F3.exe
OPTI[INFO ]: OOAnalyzer version 1.0.
OPTI[INFO ]: Reading serialized data from "/data/F3.ser".
OPTI[INFO ]: Reading serialized data took 34.7078 seconds.
OPTI[INFO ]: Partitioned 2384469 bytes, 731889 instructions, 171589 basic blocks, 10343 data blocks and 17533 functions.
...
OOAN[ERROR]: Found only 17285 functions of 17301 specifically requested for analysis.
OOAN[ERROR]: No new() methods were found.  Heap objects may not be detected.
OOAN[ERROR]: No delete() methods were found.  Object analysis may be impaired.
OPTI[WARN ]: OOAnalyzer did not perform C++ class analysis.
OPTI[INFO ]: OOAnalyzer analysis complete.

Awk prints showing that object oriented information was identified:

 105089 callParameter
  43197 callReturn
  52605 callTarget
  33022 callingConvention
      1 fileInfo
  36110 funcParameter
   6725 funcReturn
   7375 initialMemory
  44654 methodMemberAccess
   5825 noCallsAfter
   5853 noCallsBefore
     61 possibleVBTableWrite
   1105 possibleVFTableWrite
   7178 possibleVirtualFunctionCall
    446 rTTIBaseClassDescriptor
    403 rTTIClassHierarchyDescriptor
    440 rTTICompleteObjectLocator
    421 rTTITypeDescriptor
   1133 returnsSelf
   3028 thisPtrAllocation
  24989 thisPtrDefinition
   3402 thisPtrOffset
    114 thunk
     68 uninitializedReads

Ooprolog then unfortunately errors out with "Unknown message: error(system_error(initialSanityChecks))":

  [29] prolog_stack:get_prolog_backtrace(100,[frame(29,clause(<clause>(0x6034f2fde6e0),6),_25826340)|_25826328],[goal_term_depth(100)]) at /usr/local/lib/swipl/library/prolog_stack.pl:137
  [28] throw_with_backtrace(error(system_error(initialSanityChecks))) at /usr/local/share/pharos/prolog/oorules/util.pl:185
  [26] solve_internal at /usr/local/share/pharos/prolog/oorules/setup.pl:679
  [25] catch(user:solve_internal,_25826564,user:((_25826632=error(resource_error(private_table_space),_25826646)->complain_table_space(ooscript);_25826696=error(resource_error(stack),_25826710)->complain_stack_size(ooscript);true),throw(_25826742))) at /usr/local/lib/swipl/boot/init.pl:562
  [24] solve(ooscript) at /usr/local/share/pharos/prolog/oorules/setup.pl:617
  [23] psolve_no_halt('<garbage_collected>') at /usr/local/share/pharos/prolog/oorules/report.pl:23
  [22] catch(user:psolve_no_halt(stream(<stream>(0x6034f3186ec0))),_25826916,user:(print_message(error,_25826982),(globalHalt->halt(1);true))) at /usr/local/lib/swipl/boot/init.pl:562
  [21] catch_with_backtrace('<garbage_collected>','<garbage_collected>','<garbage_collected>') at /usr/local/lib/swipl/boot/init.pl:629
  [20] run_with_backtrace('<garbage_collected>') at /usr/local/bin/ooprolog:177
  [19] <meta call>
  [18] with_output_to(<stream>(0x6034f3187140),run_with_backtrace(psolve_no_halt(stream(<stream>(0x6034f3186ec0))))) <foreign>
  [17] setup_call_catcher_cleanup(user:(var('/data/F3-results.pl')->open_null_stream(<stream>(0x6034f3187140));open('/data/F3-results.pl',write,<stream>(0x6034f3187140))),user:with_output_to(<stream>(0x6034f3187140),run_with_backtrace(psolve_no_halt(stream(<stream>(0x6034f3186ec0))))),_25827334,user:close(<stream>(0x6034f3187140))) at /usr/local/lib/swipl/boot/init.pl:663
  [15] setup_call_catcher_cleanup(user:open('/data/F3-facts.pl',read,<stream>(0x6034f3186ec0)),user:setup_call_cleanup((var('/data/F3-results.pl')->open_null_stream(<stream>(0x6034f3187140));open('/data/F3-results.pl',write,<stream>(0x6034f3187140))),with_output_to(<stream>(0x6034f3187140),run_with_backtrace(psolve_no_halt(stream(<stream>(0x6034f3186ec0))))),close(<stream>(0x6034f3187140))),_25827544,user:close(<stream>(0x6034f3186ec0))) at /usr/local/lib/swipl/boot/init.pl:663
  [12] run([script('/usr/local/bin/ooprolog'),json(_25827822),ground(_25827842),rtti(true),guess(true),config(_25827902),stacklimit(200000000000),tablespace(200000000000),oorulespath(_25827962),halt(true),load_only(false),help(_25828022),facts('/data/F3-facts.pl'),results('/data/F3-results.pl'),loglevel(6)]) at /usr/local/bin/ooprolog:235
   [9] catch(user:main(['/usr/local/bin/ooprolog','--facts','/data/F3-facts.pl','--results','/data/F3-results.pl','--log-level=6']),_25828146,user:(print_message(error,_25828276),halt(1))) at /usr/local/lib/swipl/boot/init.pl:562
   [7] catch(user:main,_25828350,'$toplevel':true) at /usr/local/lib/swipl/boot/init.pl:562
   [6] catch_with_backtrace('<garbage_collected>','<garbage_collected>','<garbage_collected>') at /usr/local/lib/swipl/boot/init.pl:629

Note: some frames are missing due to last-call optimization.
Re-run your program in debug mode (:- debug.) to get more detail.
ERROR: /data/F3-facts.pl:383247:
ERROR:    Unknown message: error(system_error(initialSanityChecks))
sei-eschwartz commented 8 months ago

Hi Shane,

Can you please try to rerun without --no-semantics? This can cause problems sometimes.

It’s also very suspicious that OOAnalyzer did not find any new or delete functions. If you know how, finding those and passing them via --new-method and --delete-method would also probably help your analysis.

If theses do not help, I’ll ask you to provide some of your files so I can take a look in more detail.

Ed

From: Shane Mulcahy @.> Date: Friday, March 8, 2024 at 10:32 AM To: cmu-sei/pharos @.> Cc: Subscribed @.***> Subject: [cmu-sei/pharos] Unknown message: error(system_error(initialSanityChecks)) error from ooprolog (Issue #261) Warning: External Sender - do not click links or open attachments unless you recognize the sender and know the content is safe.

So essentially I'm trying to recover C++ class information from a 32 bit - PE-i386 format - executable compiled with MSVC. Im unsure of what version of MSVC was used to compile the executable, but I do have information on when it was compiled so the version of MSVC must be before this date -> Tue Dec 9 13:22:45 2003, so I think it must have been compiled with Visual C++ .NET 2003 / Visual C++ 7.1 or earlier:

F3.exe: file format pei-i386

executable 32 bit words

Time/Date Tue Dec 9 13:22:45 2003 Magic 010b (PE32) MajorLinkerVersion 7 MinorLinkerVersion 0 MajorOSystemVersion 4 MinorOSystemVersion 0 MajorImageVersion 0 MinorImageVersion 0 MajorSubsystemVersion 4 MinorSubsystemVersion 0 Win32Version 00000000 Subsystem 00000002 (Windows GUI)

I installed pharos via the docker image from:

I am running the docker image via:

The following is a summary of the outputs of each step. I can also provide the .exe file, .ser file, facts.pl file, results.pl file, and/or .log file if required.

Partitioning is successful:

OPTI[INFO ]: Analyzing executable: /data/F3.exe

OPTI[INFO ]: ROSE stock partitioning took 14.9721 seconds.

OPTI[INFO ]: Partitioned 731129 bytes, 244954 instructions, 56272 basic blocks, 65 data blocks and 3480 functions.

OPTI[INFO ]: Function partitioning took 677.933 seconds.

OPTI[INFO ]: Writing serialized data to "/data/F3.ser".

OPTI[INFO ]: Writing serialized data took 67.9488 seconds.

OPTI[INFO ]: Partitioned 2384469 bytes, 731889 instructions, 171589 basic blocks, 10343 data blocks and 17533 functions.

Analysis I think is successful:

OPTI[INFO ]: Analyzing executable: /data/F3.exe

OPTI[INFO ]: OOAnalyzer version 1.0.

OPTI[INFO ]: Reading serialized data from "/data/F3.ser".

OPTI[INFO ]: Reading serialized data took 34.7078 seconds.

OPTI[INFO ]: Partitioned 2384469 bytes, 731889 instructions, 171589 basic blocks, 10343 data blocks and 17533 functions.

...

OOAN[ERROR]: Found only 17285 functions of 17301 specifically requested for analysis.

OOAN[ERROR]: No new() methods were found. Heap objects may not be detected.

OOAN[ERROR]: No delete() methods were found. Object analysis may be impaired.

OPTI[WARN ]: OOAnalyzer did not perform C++ class analysis.

OPTI[INFO ]: OOAnalyzer analysis complete.

Awk prints showing that object oriented information was identified:

Ooprolog then unfortunately errors out with "Unknown message: error(system_error(initialSanityChecks))":

Note: some frames are missing due to last-call optimization.

Re-run your program in debug mode (:- debug.) to get more detail.

ERROR: /data/F3-facts.pl:383247:

ERROR: Unknown message: error(system_error(initialSanityChecks))

— Reply to this email directly, view it on GitHubhttps://github.com/cmu-sei/pharos/issues/261, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL6ZAVE6AEVSLLSEUOVCHHLYXHKXVAVCNFSM6AAAAABENACIDGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3TMMRWGY2TEMY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Shenmarukai commented 8 months ago

Ok thank you, I will attempt without --no-semantics. And for adding the new and delete functions via --new-method and --delete-method, how would I go about doing that. Would I find them manually using a disassembler and then pass them in in some format?

sei-eschwartz commented 8 months ago

Yes, you would try to identify them using a disassembler and then pass the addresses as arguments. If you can’t, do worry about it for now.

My colleague also noted that the “OOAN[ERROR]: Found only 17285 functions of 17301 specifically requested for analysis.” error is unusual. Could you upload a copy of that entire log if you have it?

From: Shane Mulcahy @.> Date: Friday, March 8, 2024 at 10:57 AM To: cmu-sei/pharos @.> Cc: Edward J Schwartz @.>, Comment @.> Subject: Re: [cmu-sei/pharos] Unknown message: error(system_error(initialSanityChecks)) error from ooprolog (Issue #261) Warning: External Sender - do not click links or open attachments unless you recognize the sender and know the content is safe.

Ok thank you, I will attempt without --no-semantics. And for adding the new and delete functions via --new-method and --delete-method, how would I go about doing that. Would I find them manually using a disassembler and then pass them in in some format?

— Reply to this email directly, view it on GitHubhttps://github.com/cmu-sei/pharos/issues/261#issuecomment-1985947722, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL6ZAVHX7J4CR2YLEUBX7BLYXHNYDAVCNFSM6AAAAABENACIDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVHE2DONZSGI. You are receiving this because you commented.Message ID: @.***>

Shenmarukai commented 8 months ago

Ok, I am testing it without --no-semantics right now. Here is the log for the previous attempt: F3-log.zip

sei-eschwartz commented 8 months ago

Can you provide the log for the fact generation step?

From: Shane Mulcahy @.> Date: Friday, March 8, 2024 at 11:14 AM To: cmu-sei/pharos @.> Cc: Edward J Schwartz @.>, Comment @.> Subject: Re: [cmu-sei/pharos] Unknown message: error(system_error(initialSanityChecks)) error from ooprolog (Issue #261) Warning: External Sender - do not click links or open attachments unless you recognize the sender and know the content is safe.

Ok, I am testing it without --no-semantics right now. Here is the log for the previous attempt: F3-log.ziphttps://github.com/cmu-sei/pharos/files/14540426/F3-log.zip

— Reply to this email directly, view it on GitHubhttps://github.com/cmu-sei/pharos/issues/261#issuecomment-1985972750, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL6ZAVEZOSC3OTWODCWDK43YXHPMBAVCNFSM6AAAAABENACIDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVHE3TENZVGA. You are receiving this because you commented.Message ID: @.***>

Shenmarukai commented 8 months ago

Here is the facts.pl: F3-facts.zip

Shenmarukai commented 8 months ago

The attempt without --no-semantics failed as well. I've include a text file of the full console session, the .log file, and the facts.pl file. It did run out of memory when analyzing so I reduced the memory limit so it wouldn't run out of memory. I'm going to try it again on my other computer that has more ram. F3-logs.zip

sei-eschwartz commented 8 months ago

Thanks, we were able to replicate the problem using your facts file. I will take a look in more detail and see what is going on.

Shenmarukai commented 8 months ago

Ok, thank you!

sei-eschwartz commented 8 months ago

These are mostly notes to myself.

reasonMergeVFTables_A(constructor, 0x673a20, 0x5006f0, 0x673a20, 0, factVFTableWrite(0x500709, 0x5006f0, 0, 0x673a20)).
Concluding mergeVFTables(0x673a20, 0x5006f0).
Retracting factObjectInObject(0x673944, 0x5006f0, 0) and asserting factObjectInObject(0x673944, 0x673a20, 0) ...
Retracting factObjectInObject(0x673944, 0x5006f0, 0x6c) and asserting factObjectInObject(0x673944, 0x673a20, 0x6c) ...
Retracting factObjectInObject(0x673944, 0x5006f0, 0xa4) and asserting factObjectInObject(0x673944, 0x673a20, 0xa4) ...
Retracting factObjectInObject(0x673944, 0x5006f0, 0xdc) and asserting factObjectInObject(0x673944, 0x673a20, 0xdc) ...
Class 0x673944 inherits from 0x673a20 at offsets 0 and 0x6c

I'm confused because clearly the same problem was already happening with 0x673944 and 0x5006f0.

Edit: Ah, because of this fact: factDerivedClass(0x673944, 0x673a20, 0). (look at insanityInheritanceTwice)

sei-eschwartz commented 8 months ago

So that fact comes from here:

% Because RTTI tells us so for a non-virtual base class. reasonDerivedClass_D(0x6fbba4, 0x6fbb88, 0x673944, 0x673a20, 0).

This is usually pretty reliable.

The other one originates here: % This rule is a special case of the reasonObjectInObject_E, that relies on the fact that it % does not matter whether the InnerClass and OuterClass are provably different or whether % they're just currently not assigned to the same class. The key observation is that the % distinction only matters when offset is non-zero, because that fact alone rules out the % possibility that the two classes are in fact the same class. reasonObjectInObject_D(0x4fb5b0, 0x5006f0, 0x6c, validMethodCallAtOffset(0x4fb62e, 0x4fb5b0, 0x5006f0, 0x6c)).

[eschwartz@pd4 f3]$ ~mwd/tmp/plogtrace.py --forward log 'factObjectInObject(0x673944, 0x673a20, 0x6c)' 
47864: Concluding factObjectInObject(0x4fb5b0, 0x5006f0, 0x6c).
543219: Merging class 0x4fb5b0 into 0x673944 ...
543225: Retracting factObjectInObject(0x4fb5b0, 0x5006f0, 0x6c) and asserting factObjectInObject(0x673944, 0x5006f0, 0x6c) ...
555075: Merging class 0x5006f0 into 0x673a20 ...
555083: Retracting factObjectInObject(0x673944, 0x5006f0, 0x6c) and asserting factObjectInObject(0x673944, 0x673a20, 0x6c) ...

Edit: Of course the merges could be wrong too...

sei-eschwartz commented 8 months ago

I have a feeling that this is caused by a thisptr offset. Can you please upload the executable? I may have a private branch of OOAnalyzer that may work on this.

Shenmarukai commented 8 months ago

Here is the executable: F3.zip

sei-eschwartz commented 8 months ago

Thanks. OK, there are two problematic functions.

0x4fb5b0 installs DerivedClass::vftable, and calls 0x5006f0 a bunch. 0x5006f0 install Base::vftable to offset 0.

Now it looks like 0x4fb5b0 is a constructor for DerivedClass, and 0x5006f0 is a constructor for BaseClass, but that may not be true.

Let's look at the reasonObjectInObject_D conclusion. It is based on the call at 0x4fb62e, in 0x4fb5b0, which calls 0x5006f0. It basically says that because one constructor calls the other at offset 0, there must be an object relationship.

Here is the decompilation of 0x4fb5b0; I called 0x5006f0 "other_problem".

undefined4 * __thiscall problem_function(undefined4 *param_1_00,int param_2) { void *local_c; undefined *puStack_8; undefined4 local_4; local_4 = 0xffffffff; puStack_8 = &LAB_00634e87; local_c = ExceptionList; ExceptionList = &local_c; other_problem(param_1_00,param_2); /* Install Derived vftable */ *param_1_00 = &DerivedClass::vftable; param_1_00[0x13] = 0xf; param_1_00[0x12] = 0; local_4 = 0; *(undefined *)(param_1_00 + 0xe) = 0; FUN_00401b20(param_2 + 0x34,0,0xffffffff); param_1_00[0x1a] = 0xf; param_1_00[0x19] = 0; local_4._0_1_ = 1; *(undefined *)(param_1_00 + 0x15) = 0; FUN_00401b20(param_2 + 0x50,0,0xffffffff); local_4._0_1_ = 2; other_problem(param_1_00 + 0x1b,param_2 + 0x6c); *(undefined *)(param_1_00 + 0x28) = *(undefined *)(param_2 + 0xa0); local_4._0_1_ = 3; other_problem(param_1_00 + 0x29,param_2 + 0xa4); local_4 = CONCAT31(local_4._1_3_,4); *(undefined *)(param_1_00 + 0x36) = *(undefined *)(param_2 + 0xd8); other_problem(param_1_00 + 0x37,param_2 + 0xdc); *(undefined *)(param_1_00 + 0x44) = *(undefined *)(param_2 + 0x110); ExceptionList = local_c; return param_1_00; }

But the same constructor is called at multiple offsets, which suggests that they are all embedded objects instead of inheritance. So maybe one of the merge conclusions is wrong.

sei-eschwartz commented 8 months ago

543219: Merging class 0x4fb5b0 into 0x673944 ...

This merges the MaybeDerivedConstructor with Derived::vftable.

reasonMergeVFTables_A(constructor, 0x673944, 0x4fb5b0, 0x673944, 0, factVFTableWrite(0x4fb5e0, 0x4fb5b0, 0, 0x673944)).

This rule basically says that because 0x4fb5e0 is a constructor, and we don't see any vftable overwrites, we can conclude that 0x4fb5e0 is part of the derived class.

We conclude that 0x4fb5e0 is a constructor because it installs a vftable before calling a constructor.

This all seems fine to me.

sei-eschwartz commented 8 months ago

One possibility is that the derived class really does inherit from the base class. But then it embeds it too?

Or, maybe there is some templated class that shares a common constructor?

sei-eschwartz commented 8 months ago

Whatever this binary is doing is kind of weird. But I think this is probably a reasonable workaround for now:

diff --git a/share/prolog/oorules/insanity.pl b/share/prolog/oorules/insanity.pl
index b99cef92..28e613a9 100644
--- a/share/prolog/oorules/insanity.pl
+++ b/share/prolog/oorules/insanity.pl
@@ -68,7 +68,8 @@ insanityInheritanceTwice(Out) :-
     % making some stupid guesses in which we inherit the first instance of a class and then
     % embed the rest.  Maybe a better fix would be to modify guessDerivedClass so that we don't
     % make the guess if there are multiple objectInObject facts.
-    factObjectInObject(A,B,O2),
+    factDerivedClass(A,B,O2),
+    %factObjectInObject(A,B,O2),
     iso_dif(O1, O2),

     Out = (

Apply that patch and then rerun the ooprolog step. I haven't run to completion yet, but it completes initial reasoning with this change at least.

Shenmarukai commented 8 months ago

Thank you! How would I apply that patch?

sei-eschwartz commented 8 months ago

Probably the easiest way is to find the insanity.pl file in the docker container and manually edit it using vi, nano, or another text editor.

Alternatively if you find the directory of insanity.pl and enter it, you should be able to run patch -p4 and paste in the patch. (I could be wrong on the 4, try 3 or 5 if it doesn't work).

Shenmarukai commented 8 months ago

Ok, I will try that.

Shenmarukai commented 8 months ago

So far it seems to be working. Its been running for while and keeps logging more info, so hopefully it runs to completion!

Shenmarukai commented 8 months ago

Thank you so much! It worked this time. I have the json output now! I really appreciate the help, this program seems really amazing, and will be really useful for me!

Shenmarukai commented 8 months ago

Here are all of the outputs from the successful run, would have included the log file but its too large: F3.zip

sei-eschwartz commented 8 months ago

Very glad to hear it!

I think probably what happened is that your program has a class like this:

struct D : public B {
  B b2;
  B b3;
};

I’ve seen this once before, but it’s very rare.