still have an issue with objects that cross page boundaries

[GoogleCodeExporter]

we still have an issue in trunk with objects that cross page boundaries. the 
problem is evident in one of scudette's images which you can download for 
testing/verification here: http://bit.ly/LTqSdq

I want to list dlls in two processes:

$ python vol.py -f ~/Desktop/win7_trial_64bit.raw --profile=Win7SP0x64 dlllist 
-p 2016,2236
Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
rundll32.exe pid:   2016
Command line : C:\Windows\System32\rundll32.exe 
shell32.dll,SHCreateLocalServerRunDll {995C996E-D918-4a8c-A302-45719A6F4EA7} 
-Embedding

Base                             Size Path
------------------ ------------------ ----
0x00000000ffe80000             0xf000 C:\Windows\System32\rundll32.exe
0x0000000076d40000           0x1ab000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000076b20000           0x11f000 C:\Windows\system32\kernel32.dll
0x000007fefcd50000            0x6b000 C:\Windows\system32\KERNELBASE.dll
0x0000000076c40000            0xfa000 C:\Windows\system32\USER32.dll
0x000007fefd7c0000            0x67000 C:\Windows\system32\GDI32.dll
0x000007fefe190000             0xe000 C:\Windows\system32\LPK.dll
Invalid Address 0x00344020, instantiating DllBase

Due to the error, the whole plugin halts before analyzing the 2nd process. What 
can we do to prevent entire plugins from stopping when they reach a case like 
this? I tested with scudette's branch and it seems to be fine there. It prints 
the following:

0x00000000ffe80000             0xf000 C:\Windows\System32\rundll32.exe
0x0000000076d40000           0x1ab000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000076b20000           0x11f000 C:\Windows\system32\kernel32.dll
0x000007fefcd50000            0x6b000 C:\Windows\system32\KERNELBASE.dll
0x0000000076c40000            0xfa000 C:\Windows\system32\USER32.dll
0x000007fefd7c0000            0x67000 C:\Windows\system32\GDI32.dll
0x000007fefe190000             0xe000 C:\Windows\system32\LPK.dll
------------------ ------------------ -

I don't feel confident that I can locate the exact change that's responsible 
for fixing this issue, but maybe scudette can tell us? 

Original issue reported on code.google.com by michael.hale@gmail.com on 28 Jun 2012 at 4:36

[GoogleCodeExporter]

Hi Michael,
   Looking at it more closely it seems to be specifically an issue with one of the modules returned by task.get_load_modules(). The module address itself is invalid. The usual convention when dereferencing an invalid pointer or struct field is to return a NoneObject, but in trunk we see this code:

> /home/scudette/projects/volatility/trunk/volatility/obj.py(195)__init__()
-> raise InvalidOffsetError("Invalid Address 0x{0:08X}, instantiating 
{1}".format(offset, self.obj_name))

This exception causes the calculation to abort. While in my branch the failure 
simply returns a NoneObject with the reason being invalid memory. Specifically 
this code trying to print in the table:

> 
/home/scudette/volatility/svn/volatility/plugins/windows/taskmods.py(147)render(
)
-> renderer.table_row(m.DllBase, m.SizeOfImage, m.FullDllName)
(Pdb) p m
[_LDR_DATA_TABLE_ENTRY _LDR_DATA_TABLE_ENTRY] @ 0x00343FF0
(Pdb) p m.DllBase
<Invalid Address 0x00344020, instantiating DllBase>
(Pdb) p m.FullDllName
<Invalid Address 0x00344038, instantiating FullDllName>

But on trunk this same thing is:
>/home/scudette/projects/volatility/trunk/volatility/plugins/taskmods.py(67)rend
er_text()
-> self.table_row(outfd, m.DllBase, m.SizeOfImage, str(m.FullDllName or ''))
(Pdb) p m
[_LDR_DATA_TABLE_ENTRY _LDR_DATA_TABLE_ENTRY] @ 0x00343FF0
(Pdb) p m.DllBase
*** InvalidOffsetError: InvalidOffsetError('Invalid Address 0x00344020, 
instantiating DllBase',)
(Pdb) p m.FullDllName
<volatility.obj.NoneObject object at 0x5a95190>
(Pdb) p m.FullDllName.reason
'Invalid Address 0x00344038, instantiating FullDllName'

So dereferencing the dlls name does return a NoneObject as it should, but the 
DllBase raises which is inconsistent.

This also explains the last row of -------- in the output of my branch - 
invalid NoneObjects are rendered as ------- by the renderer.

Original comment by scude...@gmail.com on 28 Jun 2012 at 5:10

[GoogleCodeExporter]

Thanks Scudette, that makes sense. I marked this as needing a fix asap, but I'm 
afraid I may have to rely on you or Ikelos for an actual patch. 

When comparing the two obj.py files, trunk's BaseObject.__init__ and your 
BaseObject.__init__ both raise InvalidOffsetError:

http://code.google.com/p/volatility/source/browse/trunk/volatility/obj.py#195
http://code.google.com/p/volatility/source/browse/branches/scudette/volatility/o
bj.py#245

So no behavior change there. Trunk's obj.Object already catches 
InvalidOffsetError and returns NoneObject, which I think should give the same 
behavior as your obj.Profile.Object:

http://code.google.com/p/volatility/source/browse/trunk/volatility/obj.py#172
http://code.google.com/p/volatility/source/browse/branches/scudette/volatility/o
bj.py#1383

The only other places I see that you catch InvalidOffsetError is 
obj.Pointer.__add__, obj.Array.__getitem__, obj.ListArray.__iter__ (none of 
which seem related).  

So hmm, you must be returning NoneObject from somewhere that I haven't found 
yet? 

Original comment by michael.hale@gmail.com on 28 Jun 2012 at 6:44

• Added labels: Milestone-2.1.x

[GoogleCodeExporter]

Hi Michael,
  Yeah you are right - this is actually a very interesting problem :-). So I checked again the backtrace for trunk and as you can see the problem happens when we try to dereference the .DllBase member of the invalid pointer:

> /home/scudette/projects/volatility/trunk/volatility/obj.py(753)m()
-> result = cls(offset = offset, vm = self.obj_vm, parent = self, name = attr, 
native_vm = self.obj_native_vm)

Now cls is actually a compiled partial object and so it is very hard to debug 
(because its opaque). I did some benchmarking and I could not find a 
significant performance advantage to using functools.partial over the old 
Curry() implementation so in my branch I put the old implementation back:
http://code.google.com/p/volatility/source/browse/branches/scudette/volatility/o
bj.py#51

The advantage with this implementation is that its much easier to debug because 
you can see exactly what is going to be called and how. So for debugging I put 
the old implementation in trunk temporarily. This is the difference between my 
branch and trunk:

> /home/scudette/projects/volatility/trunk/volatility/obj.py(753)m()
-> result = cls(offset = offset, vm = self.obj_vm, parent = self, name = attr, 
native_vm = self.obj_native_vm)
(Pdb) cls
<volatility.obj.Curry object at 0x40df450>
(Pdb) cls.target
<class 'volatility.obj.Pointer'>

While in my branch:

> /home/scudette/volatility/svn/volatility/obj.py(905)m()
-> profile=self.obj_profile, context=self.obj_context)
(Pdb) cls
<volatility.obj.Curry object at 0x39a7150>
(Pdb) cls.target
<bound method Win7SP1x64.Object of 
<volatility.plugins.overlays.windows.win7.Win7SP1x64 object at 0x30aa510>>

So whats happening here is that when the profile is compiled, the CType object 
is initialized with a curried function for each member, but there seems to be a 
bug in trunk's Profile compiler. The bug is that trunk tries to instantiate the 
class directly, while my branch always uses the Object() factory method to 
instantiate all classes. The difference here is because the Object() factory 
already has an exception handler for invalid addresses and it then returns a 
NoneObject:
http://code.google.com/p/volatility/source/browse/branches/scudette/volatility/o
bj.py#1383

(Trunk is similar too).

The scudette branch has had a big refactor for the object compiler and also 
updated the vtypes language to be more consistent. I suspect to fix this in 
trunk would require a lot of refactoring to the object type system (maybe its 
just easier to backport the scudette branch one?).

Anyway the quick fix right now for this issue is to simply wrap the cls() 
instantiation in the m() method and catch the exception there, returning a 
NoneObject.

Original comment by scude...@gmail.com on 28 Jun 2012 at 8:13

[GoogleCodeExporter]

MHL, please try the attached patch.  It's a proof of concept that covers the 
majority of scenarios and can be extended to others should the need arise.

Original comment by mike.auty@gmail.com on 1 Jul 2012 at 4:32

Attachments:

• volatility-object-factory-class-builder.patch

[GoogleCodeExporter]

Hey Mike, yessir that seemed to do the job. 

Original comment by michael.hale@gmail.com on 1 Jul 2012 at 11:33

[GoogleCodeExporter]

This issue was closed by revision r1988.

Original comment by mike.auty@gmail.com on 8 Jul 2012 at 1:16

• Changed state: Fixed