OR'd flags in vtypes

[GoogleCodeExporter]

Hey guys, quick question. We can have BitFields and Enumerations in vtypes. I'm 
looking for something similar to Enumerations but for flags that are OR'd 
together. For example, right now I define service types like this:

SERVICE_TYPES = dict(
  SERVICE_KERNEL_DRIVER = 0x01,
  SERVICE_FILE_SYSTEM_DRIVER = 0x02,
  SERVICE_WIN32_OWN_PROCESS = 0x10,
  SERVICE_WIN32_SHARE_PROCESS = 0x20,
  SERVICE_INTERACTIVE_PROCESS = 0x100,
)

Then use a function like the following:

def get_flags(flags, val):
    return [f for f in flags if flags[f] & val]

Then get_flags(SERVICE_TYPES, 0x110) will return "SERVICE_WIN32_OWN_PROCESS | 
SERVICE_INTERACTIVE_PROCESS"

Is it possible to design something like this instead:

'_SERVICE_RECORD' : [ None, {
    'ServiceType' : [ 0, ['OrFlags', dict(target = 'long', choices = {1: 'SERVICE_KERNEL_DRIVER', 2: 'SERVICE_FILE_SYSTEM_DRIVER', 0x10: 'SERVICE_WIN32_OWN_PROCESS', 0x20: 'SERVICE_WIN32_SHARE_PROCESS', 0x100: 'SERVICE_INTERACTIVE_PROCESS'})]],
}]

Of course OrFlags isn't a real type. I think what I want is something similar 
to what we do with the bitmap and maskmap Flags here:

xpsp2overlays['_MMVAD_SHORT'] = [ None, {
    'Flags': [ None, ['Flags',
  {'bitmap': {
    'PhysicalMapping': 0x13,
    'ImageMap': 0x14,
    'UserPhysicalPages': 0x15,
    'NoChange': 0x16,
    'WriteWatch': 0x17,
    'LargePages': 0x1D,
    'MemCommit': 0x1E,
    'PrivateMemory': 0x1f,
    },
  'maskmap': {
    'CommitCharge' : [0x0, 0x13],
    'Protection' : [0x18, 0x5],
    }
  } ] ],
    } ]

Is it possible to do something similar for the OR'd flags? If its already 
implemented, is there an example you can point me to?

Thanks!

Original issue reported on code.google.com by michael.hale@gmail.com on 30 Sep 2011 at 6:12

[GoogleCodeExporter]

Michael,
  You can create your own class to handle this, e.g.:

class OrFlags(obj.NATIVE_TYPE):
   def __init__(self, theType, offset, vm, parent = None,
                target = "long", choices=None):
      self.choices = choices
      ...  call baseclass ...

   def v(self):
     return [f for f in self.choices if self.choices[f] & val]

The in your plugin add this to the profile's object classes

AbstractWindows.object_classes['OrFlags'] = OrFlags

and overlay like you had before. Remember to call profile.add_types() to 
recompile the vtypes.

You should be able to just do SERVICE_TYPES.v() to get the breakdown.

Original comment by scude...@gmail.com on 30 Sep 2011 at 7:22

[GoogleCodeExporter]

On second look are you not really looking for a bitmap here? Remember that the 
bitmap value is the single bit position which selects that value:

bitmap = dict(
  SERVICE_KERNEL_DRIVER = 0x01,
  SERVICE_FILE_SYSTEM_DRIVER = 0x02,
  SERVICE_WIN32_OWN_PROCESS = 0x4,
  SERVICE_WIN32_SHARE_PROCESS = 0x6,
  SERVICE_INTERACTIVE_PROCESS = 0x9,
)

Currently the str() method separates the values with a ,. It might be a good 
idea to change the .v() method to return a list of strings.

Original comment by scude...@gmail.com on 30 Sep 2011 at 7:30

[GoogleCodeExporter]

Thanks Scudette, you're a huge help. I'm about the try the 
OrFlags(obj.NativeType) method. I don't think bitmap is what I need after 
giving that a shot. Here's the bitmap method:

'Type' : [ 0x28, ['Flags', {'bitmap': {'SERVICE_KERNEL_DRIVER': 0x1, 
'SERVICE_FILE_SYSTEM_DRIVER': 0x2, 'SERVICE_WIN32_OWN_PROCESS': 0x10, 
'SERVICE_WIN32_SHARE_PROCESS': 0x20, 'SERVICE_INTERACTIVE_PROCESS': 0x100}}]],

When the actual value is 0x1, 0x10, 0x20, 0x110, or 0x120, the result of 
printing object.Type or str(object.Type) is just an empty string. If the actual 
value is 0x2, it prints "SERVICE_KERNEL_DRIVER". Does that make sense? Does it 
look like I'm using the bitmap method correctly? 

Anyway, I'll let you know how the OrFlags(obj.NativeType) method goes. That for 
sure should work, just wanted to try and avoid writing the extra code if not 
necessary. 

Original comment by michael.hale@gmail.com on 30 Sep 2011 at 11:48

[GoogleCodeExporter]

Alright, the OrFlags(obj.NativeType) method worked. Here's what I did just for 
documentation purposes. 

The type is like this:

'Type' : [ 0x28, ['OrFlags', dict(target = 'long', choices = {1: 
'SERVICE_KERNEL_DRIVER', 2: 'SERVICE_FILE_SYSTEM_DRIVER', 0x10: 
'SERVICE_WIN32_OWN_PROCESS', 0x20: 'SERVICE_WIN32_SHARE_PROCESS', 0x100: 
'SERVICE_INTERACTIVE_PROCESS'})]],

The class looks like this:

class OrFlags(obj.NativeType):
    "Classic OR flag handler"

    def __init__(self, theType = None, offset = 0, vm = None, parent = None,
                 choices = None, name = None, target = "unsigned long",
                 **args):
        self.choices = choices or {}
        self.target_obj = obj.Object(target, offset = offset, vm = vm, parent = parent)
        obj.NativeType.__init__(self, theType, offset, vm, parent, **args)

    def v(self):
        return self.target_obj.v()

    def __str__(self):
        value = self.v()
        try:
            return self.choices[value]
        except KeyError:             
            return ' | '.join([self.choices[f] for f in self.choices.keys() if f & value])

In the plugin, I added the class like this:

addr_space.profile.object_classes['OrFlags'] = OrFlags

So, Scudette, my two questions for you are:

1) Do you agree that a separate obj.NativeType like OrFlags is required for 
this functionality, as opposed to using bitmaps with the Flags class?

2) If yes, do you think we should include OrFlags in basic.py for use by other 
plugins (for example it could be used instead of this: 
http://code.google.com/p/volatility/source/browse/trunk/volatility/win32/rawreg.
py#137)

Thanks again...

Original comment by michael.hale@gmail.com on 1 Oct 2011 at 1:56

[GoogleCodeExporter]

Argh sorry I am lousy doing maths in my head. So the Flags class takes a bitmap 
parameter which is a dict. The keys of the DICT are the flag values and the 
values are the bit position which signifies the flag is set. The example you 
gave is:

SERVICE_TYPES = dict(
  SERVICE_KERNEL_DRIVER = 0x01,
  SERVICE_FILE_SYSTEM_DRIVER = 0x02,
  SERVICE_WIN32_OWN_PROCESS = 0x10,
  SERVICE_WIN32_SHARE_PROCESS = 0x20,
  SERVICE_INTERACTIVE_PROCESS = 0x100,
)

so the SERVICE_KERNEL_DRIVER is on when the 0'th bit is on (i.e. & 0x1). The 
numbers are related via a logarithmic relation (e.g. math.log(0x100, 2) = 8 
etc).

So the bitmap should be:

bitmap = dict(
  SERVICE_KERNEL_DRIVER = 0x0,
  SERVICE_FILE_SYSTEM_DRIVER = 0x01,
  SERVICE_WIN32_OWN_PROCESS = 0x4,
  SERVICE_WIN32_SHARE_PROCESS = 0x5,
  SERVICE_INTERACTIVE_PROCESS = 0x8,
)

It might be easier to just add an additional parameter to the Flags class so it 
can be initialized in the way you prefer to think about it (i.e. with mask 
types). Although it actually does not make any practical sense to specify flags 
in term of non whole powers of 2 so I would doubt there are instances of flags 
which really work this way.

Original comment by scude...@gmail.com on 1 Oct 2011 at 11:58

[GoogleCodeExporter]

Alright, I'll go with the Flags:bitmap method using math.log versions of the 
actual constants. It's a little less readable than I'd hoped for (since you'd 
expect to see the same constants used in windows source code or in C header 
files), but at least this way we don't have to change anything. Final version 
just for documentation purposes is:

    'Type' : [ 0x28, ['Flags', {'bitmap': {'SERVICE_KERNEL_DRIVER': 0, 'SERVICE_FILE_SYSTEM_DRIVER': 1, 'SERVICE_WIN32_OWN_PROCESS': 4, 'SERVICE_WIN32_SHARE_PROCESS': 5, 'SERVICE_INTERACTIVE_PROCESS': 8}}]],

Thanks guys!

Original comment by michael.hale@gmail.com on 20 Oct 2011 at 12:48

• Changed state: WontFix