callable offset in struct is not intuitive

[GoogleCodeExporter]

Gleeda wanted to specify a member in the struct which has an offset which 
depends on another field in the same struct. This can usually be achieved by 
adding a lambda to the offset field. However the current code expects the 
lambda to return an absolute offset rather than the offset from the start of 
the struct which is more usual for a constant. This is confusing.

To repeat this bug just change e.g.:
 'UniqueProcessId' : [ 0x84, ['pointer', ['void']]],

To:

 'UniqueProcessId' : [ lambda x: 0x84, ['pointer', ['void']]],

But this will not work because currently volatility is expecting:

 'UniqueProcessId' : [ lambda x: x.obj_offset +0x84, ['pointer', ['void']]],

Which is confusing.

We can either document it or apply the following patch

--- volatility/obj.py   (revision 1058)
+++ volatility/obj.py   (working copy)
@@ -738,7 +738,7 @@
         if callable(offset):
             ## If offset is specified as a callable its an absolute
             ## offset
-            offset = int(offset(self))
+            offset = int(offset(self)) + int(self.obj_offset)
         else:
             ## Otherwise its relative to the start of our struct
             offset = int(offset) + int(self.obj_offset)

Original issue reported on code.google.com by scude...@gmail.com on 19 Aug 2011 at 9:01

[GoogleCodeExporter]

So this turns out to break the many places where variables are aliased. For 
example:

volatility/plugins/overlays/windows/xp_sp3_x86.py:

xpsp3overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset

We just alias Flags to have the same offset as "u". So these will all need to 
change to:

xpsp3overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: (x.u.obj_offset - 
x.obj_offset)

There are about a dozen places where we use this. If anyone else has private 
vtypes that use this kind of aliasing they will also break. What shall we do?

Original comment by scude...@gmail.com on 20 Aug 2011 at 7:19

[GoogleCodeExporter]

Hmmmmm, well, we could check whether the returned offset is less than 
self.obj_offset, and if so add self.obj_offset.  We may run into problems if 
people want to access just a little in front of a struct, but for most 
circumstances this will work.

Another option would be to check whether the value is less than the structure's 
size, but that again depends on structures having the right maximum size set 
(which I can see leading to people setting arbitrarily large sizes when they 
don't know).

Finally, we could create another (read-only) helper property that gives the 
relative offset of a structure if it has a parent, and then change all callable 
offsets to relative, and update any instances of x.u.obj_offset to 
x.u.obj_rel_offset.  This would be an API break though...

Original comment by mike.auty@gmail.com on 21 Aug 2011 at 9:45

[GoogleCodeExporter]

I have implemented a better way for aliasing members by allowing callables as 
direct struct members. For example this aliases the x.u member as Flags:

xpsp3overlays['_MMVAD_SHORT'][1]['Flags'] = lambda x: x.u

We should remove all the old aliasing to the new technique and go ahead and fix 
this bug.

Original comment by scude...@gmail.com on 30 Sep 2011 at 7:37

[GoogleCodeExporter]

update - maybe in x64 branch and def in linux, but we'll keep the issue open 
until confirming which branch(s) it exists in and which ones still need to be 
fixed (if any)

Original comment by michael.hale@gmail.com on 15 Dec 2011 at 9:29

[GoogleCodeExporter]

I've checked out both of the linux branches and the windows x64 branch, but 
none of these work with the plugin I wrote :-(  The plugin does work when I 
apply the first patch however.  Here's an example of a struct below.  The 
problem occurs with the last two members which rely on the value of a previous 
member to get their offset:

    'THE_STRUCT': [ 0x100, {
        'F_Offset': [0x14, ['unsigned short']],
        'Item1': [0x20, ['unsigned long long']],
        'Item2': [0x28, ['unsigned short']],
        # lambda x : x.F_Offset doesn't work for me for the offset :'-( w/o fixup of issue 138 
        'A': [lambda x : x.F_Offset, ['A_TYPE']], 
        'B': [lambda x : x.F_Offset, ['B_TYPE']],
     }],

How should I create this struct such that it will work as intended?

Original comment by jamie.l...@gmail.com on 29 Dec 2011 at 3:32

[GoogleCodeExporter]

Jamie, I think the member must be a direct callable. If you are trying to 
instantiate another object at the offset given by F_Offset, I think this should 
be:

'THE_STRUCT': [ 0x100, {
        'F_Offset': [0x14, ['unsigned short']],
        'Item1': [0x20, ['unsigned long long']],
        'Item2': [0x28, ['unsigned short']],
        'A': lambda x : obj.Object("A_TYPE", x.F_Offset, address_space=x.obj_vm), 
     }],

But I will have to check later this evening.

Original comment by scude...@gmail.com on 29 Dec 2011 at 4:30

[GoogleCodeExporter]

Ok, that makes sense, except I think it should be:

   'A': lambda x : obj.Object("A_TYPE", offset=x.F_Offset, vm=x.obj_vm), 

Now suppose "A" is something a little more complicated:

   'A': [lambda x : x.F_Offset, ['array', lambda x : x.Count, ['unsigned short']]],

Original comment by jamie.l...@gmail.com on 29 Dec 2011 at 6:33

[GoogleCodeExporter]

You can do it the same way, its just that you really want an array in
this case. The array object has this constructor (we probably need a
docstring there):

def __init__(self, theType, offset, vm = None, parent = None,
                 count = 1, targetType = None, target = None, name = None):

So you need to supply a count (for the length of the array), and
either a targetType or a callable as target which will be instantiated
over each offset in the array (the size of the object is used to
advance the array).

So you can do this:
'A': lambda x: obj.Object("Array", x.F_Offset, self.addrspace, count=x.Count,
                                        target=obj.Curry(obj.Object,
"unsigned int"))

Or this:

'A': lambda x: obj.Object("Array", x.F_Offset, self.addrspace, count=x.Count,
                                       target=lambda **kw:
obj.Object("unsigned int", **kw))

Or this:

'A': lambda x: obj.Object("Array", x.F_Offset, self.addrspace, count=x.Count,
                                       targetType"unsigned int")

Or, for more complex things you can use a class override:

Class Foobar(obj.CType):

   def _A(self, attr):
       return obj.Object("Array", self.F_Offset, self.addrspace,
count=self.Count,
                                       targetType"unsigned int")

IMHO these are also more readable than the original vtype encoding method.

Hope this helps,
Michael.

Original comment by scude...@gmail.com on 29 Dec 2011 at 11:25

[GoogleCodeExporter]

(Except as of a little while ago, and r1181, the final example would read:

Class Foobar(obj.CType):

   @property
   def A(self):
       return obj.Object("Array", self.F_Offset, self.addrspace, 
count=self.Count,
                                       targetType"unsigned int")

)  5;)

Original comment by mike.auty@gmail.com on 29 Dec 2011 at 11:30

[GoogleCodeExporter]

Am I going crazy, or is there absolutely no need for 8 of the 12 MMVAD 
u.obj_offset overrides?  They all should get there from the xpsp2 overlay, 
shouldn't they?

Original comment by mike.auty@gmail.com on 31 Dec 2011 at 9:45

[GoogleCodeExporter]

Ok, member aliasing has hit the trunk (r1280), and @property is the way to 
handle new properties.  Can this bug be closed out?

Original comment by mike.auty@gmail.com on 23 Jan 2012 at 2:54

[GoogleCodeExporter]

Yes, that's fine to close this out.  Thanks for the update :-)

Original comment by jamie.l...@gmail.com on 23 Jan 2012 at 3:22

• Changed state: Fixed