search

AndrewLabs / volatility

Automatically exported from code.google.com/p/volatility
0 stars 0 forks source link

MMVAD_FLAGS members deleted from xp_sp2_x86_vtypes.py? #144

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Hmm this is strange. If you take a look at the following MMVAD_FLAGS definition 
from xp_sp2_x86_vtypes.py you'll notice all its members have been deleted:

http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overl
ays/windows/xp_sp2_x86_vtypes.py#2078

Or for convenience, here's what you'll see:

  '_MMVAD_FLAGS' : [ 0x4, {
} ],

I figure one of two things happened:

1) someone must have done it while working on the overlays in xp_sp2_x86.py
2) BDG's PDB parser originally didn't work with bitfields and never 
auto-generated the structures (but it does now)

http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overl
ays/windows/xp_sp2_x86.py#218

Once again for convenience:

xpsp2overlays['_MMVAD_SHORT'] = [ None, {
    'Flags': [ None, ['Flags',
  {'bitmap': {
    'PhysicalMapping': 0x13,
    'ImageMap': 0x14,
    'UserPhysicalPages': 0x15,
    'NoChange': 0x16,
    'WriteWatch': 0x17,
    'LargePages': 0x1D,
    'MemCommit': 0x1E,
    'PrivateMemory': 0x1f,
    },
  'maskmap': {
    'CommitCharge' : [0x0, 0x13],
    'Protection' : [0x18, 0x5],
    }
  } ] ],
    } ]

I ran into an issue when looking for a clean way to test if the PrivateMemory 
bit was set. As you may know, vadinfo prints the flags like this:

outfd.write("Flags: {0}\n".format(vad.Flags))

And it comes out looking like this:

Flags: MemCommit, NoChange, PrivateMemory

What I would like to be able to do is type "if vad.Flags.PrivateMemory == 1" 
but instead the only way I found was doing this instead:

if "PrivateMemory" in format(vad.Flags):

Of course, that is not the desired method. I found that if we add the 
MMVAD_FLAGS structure members back into xp_sp2_x86_vtypes.py...for example:

  '_MMVAD_FLAGS' : [ 0x4, {
    'CommitCharge' : [ 0x0, ['BitField', dict(start_bit = 0, end_bit = 19)]],
    'PhysicalMapping' : [ 0x0, ['BitField', dict(start_bit = 19, end_bit = 20)]],
    'ImageMap' : [ 0x0, ['BitField', dict(start_bit = 20, end_bit = 21)]],
    'UserPhysicalPages' : [ 0x0, ['BitField', dict(start_bit = 21, end_bit = 22)]],
    'NoChange' : [ 0x0, ['BitField', dict(start_bit = 22, end_bit = 23)]],
    'WriteWatch' : [ 0x0, ['BitField', dict(start_bit = 23, end_bit = 24)]],
    'Protection' : [ 0x0, ['BitField', dict(start_bit = 24, end_bit = 29)]],
    'LargePages' : [ 0x0, ['BitField', dict(start_bit = 29, end_bit = 30)]],
    'MemCommit' : [ 0x0, ['BitField', dict(start_bit = 30, end_bit = 31)]],    
    'PrivateMemory' : [ 0x0, ['BitField', dict(start_bit = 31, end_bit = 32)]],
} ],

Now we can test individually like vad.u.VadFlags.PrivateMemory. It also doesn't 
"break" vadinfo's printing statements since that all comes from the overlay. 

So in summary, I would like to add the MMVAD_FLAGS structure back into 
xp_sp2_x86_vtypes.py for the following reasons:

1) It won't change how existing plugins access the flags
2) It shouldn't have been removed in the first place if vtypes are supposed to 
be 100% auto-generated
3) It provides an easier way to test flag bits instead of relying on format 
strings etc

Just FYI, MMVAD_FLAGS2 is also missing from xpsp2 vtypes, so we should add that 
as well.

Original issue reported on code.google.com by michael.hale@gmail.com on 29 Sep 2011 at 4:26

GoogleCodeExporter commented 9 years ago 
More info...

The MMVAD_FLAGS, MMVAD_FLAGS2, and MMVAD_FLAGS3 are empty for the following 
profiles:

vista_sp0_x86_vtypes.py
vista_sp1_x86_vtypes.py
vista_sp2_x86_vtypes.py

win7_sp0_x86_vtypes.py
xp_sp3_x86_vtypes.py

The structures are non-empty (and thus OK) for all others.

Original comment by michael.hale@gmail.com on 29 Sep 2011 at 4:37

GoogleCodeExporter commented 9 years ago 
Here's a patch to re-apply the MMVAD_FLAGS fields to the profiles that define 
them as empty. Thoughts?

Original comment by michael.hale@gmail.com on 29 Sep 2011 at 4:57

Attachments:

GoogleCodeExporter commented 9 years ago 
Michael, please do not apply the patch to the vtypes directly since they are 
supposed to be autogenerated. Can this be applied to the overlays?  Through 
overlay inheritance they should apply to all oses then. Do these flags change 
much between versions?

Original comment by scude...@gmail.com on 30 Sep 2011 at 8:13

GoogleCodeExporter commented 9 years ago 
> they are supposed to be autogenerated

I think I can explain this now. At the time the XP SP2 and XP2 profiles were 
autogenerated, the autogenerating scripts didn't support BitFields. See  
http://code.google.com/p/pdbparse/source/detail?r=63. Thus any structures 
requiring BitFields in profiles generated before Feburary 4, 2011 are empty. If 
you autogenerate the profiles today, you'll get the right results. So the big 
question is...when a major change like this is made to the autogenerating 
scripts, do we re-autogenerate all profiles? 

Just to clear up something else, the vista and win7 sp0 that are also missing 
the BitFields were probably generated after Feb 4 2011. My guess there is that 
whoever generated them (may have even been me) used an old version of the 
autogenerating scripts.

Original comment by michael.hale@gmail.com on 30 Sep 2011 at 9:29

GoogleCodeExporter commented 9 years ago 
Oops by "XP SP2 and XP2" I meant "XP SP2 and SP3"

Original comment by michael.hale@gmail.com on 30 Sep 2011 at 9:32

GoogleCodeExporter commented 9 years ago 
Ah yeah i see. That makes sense. If we feel that profiles need to be 
re-generated because pdbparse is updated, I think we should do this and update 
the repository.  Do you have all the symbol packages downloaded already?

Original comment by scude...@gmail.com on 30 Sep 2011 at 9:58

GoogleCodeExporter commented 9 years ago 
Yeah how about I re-autogenerate all the profiles this weekend and send them 
around. You guys can take a look, do some sanity checks with the vtype diffing 
script and if it looks good, then we can apply them.

Original comment by michael.hale@gmail.com on 30 Sep 2011 at 10:54

GoogleCodeExporter commented 9 years ago 
Alright here are the raw profiles. They haven't been "reduced" with the vtype 
diffing script yet. 

One thing I noticed that we could fix for consistency purposes is the name of 
the main dictionary in each profile. We seem to have a big variety which can be 
confusing:

$ grep "_types = {" *
vista_sp0_x86_vtypes.py:ntkrnlmp_types = {
vista_sp1_x86_vtypes.py:ntkrnlmp_types = {
vista_sp2_x86_vtypes.py:ntkrnlmp_types = {
win2k3_sp0_x86_vtypes.py:ntoskrnl_types = {
win2k3_sp1_x86_vtypes.py:ntoskrnl_types = {
win2k3_sp2_x86_vtypes.py:ntoskrnl_types = {
win7_sp0_x86_vtypes.py:ntkrpamp_types = {
win7_sp1_x86_vtypes.py:ntkrnlmp_types = {
xp_sp2_x86_vtypes.py:xpsp2_types = {
xp_sp3_x86_vtypes.py:ntoskrnl_types = {

I don't think it matters if we auto-generate the profile from ntoskrnl.exe, 
ntkrpamp.exe, or ntkrnlmp.exe, but please let me know if you think otherwise.

Original comment by michael.hale@gmail.com on 3 Oct 2011 at 12:29

Attachments:

GoogleCodeExporter commented 9 years ago 
Hey guys, 

Anyone given some thought to this? I did a little testing today and wanted to 
share some results. Mainly I wanted to see if it matters which version of the 
NT module we use to auto-generate a profile. 

So for the test I took ntoskrnl.exe and ntkrnlpa.exe from a 2k8sp2 machine. I 
downloaded the PDB for both using pdbparse's symcheck.py:

$ python symchk.py ntoskrnl.exe 
Trying 
http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/0F9B6FF39CAB48838BFF8807
77AC7D0E2/ntkrnlmp.pd_
Connected. Downloading data...
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 
100%
Saved symbols to ntkrnlmp.pd_
Extracting cabinet: ntkrnlmp.pd_
  extracting ntkrnlmp.pdb
All done, no errors.

$ python symchk.py ntkrnlpa.exe 
Trying 
http://msdl.microsoft.com/download/symbols/ntkrpamp.pdb/109FACEC7E244C8FAC6D1914
57B5C7022/ntkrpamp.pd_
Connected. Downloading data...
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 
100%
Saved symbols to ntkrpamp.pd_
Extracting cabinet: ntkrpamp.pd_
  extracting ntkrpamp.pdb
All done, no errors.

As you can see, the PDB for this ntoskrnl.exe is ntkrnlmp.pdb and the PDB for 
this ntkrnlpa.exe is ntkrpamp.pdb. 

I use the two PDBs with pdbparse to generate the python files:

$ python tpi_vtypes.py ntkrnlmp.pdb > ntkrnlmp_vtypes.py
$ python tpi_vtypes.py ntkrpamp.pdb > ntkrpamp_vtypes.py

Then I wrote a script to compare the structure names, structure sizes, member 
names, and member offsets. The script is attached - its messy, I only intended 
on using it this once. 

Anyway, here is a snippet of the results:

$ python compare.py 

V1 only
  __unnamed_1f16
  __unnamed_149c
  __unnamed_15fb
  __unnamed_149f
  __unnamed_141c
...
  _MMPTE_HIGHLOW
  __unnamed_1457
  __unnamed_1455
V2 only
  __unnamed_1fe9
  __unnamed_1f15
  __unnamed_1f11
  __unnamed_15fc
  __unnamed_149d
Size mismatch for _MMPTE_LIST 8 -vs- 4
Size mismatch for _MMPTE 8 -vs- 4
Size mismatch for _MMPTE_PROTOTYPE 8 -vs- 4
Size mismatch for _SEGMENT 56 -vs- 48
Size mismatch for _MMPTE_SUBSECTION 8 -vs- 4
v1 only members in __unnamed_1cd0
  Length
  Reserved
v2 only members in __unnamed_1cd0
  Length64
Offset mismatch for _MM_SESSION_SPACE member PoolTrackBigPagesSize 0x1e88 -vs- 
0x1e78
Offset mismatch for _MM_SESSION_SPACE member PoolBigEntriesInUse 0x1e40 -vs- 
0x1e30
Offset mismatch for _MM_SESSION_SPACE member PoolTrackTableExpansion 0x1e7c 
-vs- 0x1e6c
Offset mismatch for _MM_SESSION_SPACE member PagedPoolPdeCount 0x1e44 -vs- 
0x1e34
Offset mismatch for _MM_SESSION_SPACE member PoolTrackBigPages 0x1e84 -vs- 
0x1e74
Offset mismatch for _MM_SESSION_SPACE member SessionPoolPdes 0x1e8c -vs- 0x1e7c
Offset mismatch for _MM_SESSION_SPACE member DynamicSessionPdeCount 0x1e4c -vs- 
0x1e3c
Offset mismatch for _MM_SESSION_SPACE member SystemPteInfo 0x1e50 -vs- 0x1e40
Offset mismatch for _MM_SESSION_SPACE member PoolTrackTableExpansionSize 0x1e80 
-vs- 0x1e70
Offset mismatch for _MM_SESSION_SPACE member SessionPteLock 0x1e20 -vs- 0x1e10
Offset mismatch for _MM_SESSION_SPACE member SpecialPoolPdeCount 0x1e48 -vs- 
0x1e38
v1 only members in __unnamed_1cc8
  Vector
  Affinity
  Level
v2 only members in __unnamed_1cc8
  Start
  Length
  Reserved
Size mismatch for _MMPFN 28 -vs- 24
Size mismatch for _MMPTE_TRANSITION 8 -vs- 4
Size mismatch for __unnamed_1bde 20 -vs- 12

So this is good and bad news to me (not sure what your take will be). Its good 
because none of the core structures are different (like eprocess, ethread, 
etc). However its bad because there clearly are differences and unless we 
detect on the fly which version of the NT module is loaded in the memory dump, 
we won't know exactly which PDB is the "correct" one. 

Although I said no core structures are different, it doesn't mean the changed 
ones are 100% insignificant. For example the MMPTE is going to be needed for 
solving Issue #143 (though perhaps we can use overlays/fixups if necessary). 

So what are your thoughts? Should we pick one NT module (ntoskrnl.exe? if not 
which one) and stick with it, or continue to generate profiles from any NT 
module and hope the differences don't become a big deal?

Original comment by michael.hale@gmail.com on 11 Oct 2011 at 9:36

Attachments:

GoogleCodeExporter commented 9 years ago 
After speaking with LabarumXP tonight, I decided to submit this patch for your 
review. Its a 31,000+ line patch replacing the existing profiles with profiles 
auto-generated using the newest version of pdbparse. 

Here are some things you need to know:

* I have not yet condensed them with vtype_diff.py. I think we should do that 
in the "next round" of changes.

* I renamed all of the ntkrnlmp_types, ntoskrnl_types, xpsp2_types, and 
ntkrpamp_types to a consistent "nt_types". Despite the fact that small things 
differ between NT* modules (as described in previous comments), it has 
obviously never caused a problem in the life of Volatility so we don't believe 
it will be an issue in the future. And if it is, we can cope by using overlays. 

* I applied the pointer64 native type (issue 105) to the win7_sp0_x86 profile 
(previously we thought it started with win7_sp1_x86 but that's just because we 
had been using a profile for win7_sp0_x86 generated using an older pdbparse)

* After applying this patch, if desired, we can delete the manually created 
overlay for _POOL_HEADER in xp_sp2_x86.py (since the auto-generated version is 
sufficient)

After the patch, we'll be able to still print things like vad.Flags due to the 
overlays in xp_sp2_x86.py *and* we'll be able to test individual bits by using 
vad.u.VadFlags.PrivateMemory == 1 (that's just an example, the same obviously 
applies to hundreds of other structures that now use BitFields and Enumerations 
correctly). 

So what I would ask of you all is to start using this patch and the new 
profiles immediately (the longer they're in place and tested before the next 
release, the better). Also, Ikelos, I could use your help applying the 
vtype_diff.py changes once time comes for that.

Original comment by michael.hale@gmail.com on 29 Oct 2011 at 5:27

Attachments:

GoogleCodeExporter commented 9 years ago 
Ah I've just noticed that for a long time, we've either been printing vad flags 
correctly by strict coincidence (in the case of CommitCharge and Protection) or 
we've been printing them incorrectly (in the case of all others). 

Its because the overlay in xp_sp2_x86.py (shown in the first comment for this 
issue) is carried through to Windows 7 (and all OS in between). As you can see, 
the overlay we're using is based on the XPSP2 definition, which is represented 
correctly in the new profiles I generated. 

  '_MMVAD_FLAGS' : [ 0x4, {
    'CommitCharge' : [ 0x0, ['BitField', dict(start_bit = 0, end_bit = 19)]],
    'PhysicalMapping' : [ 0x0, ['BitField', dict(start_bit = 19, end_bit = 20)]],
    'ImageMap' : [ 0x0, ['BitField', dict(start_bit = 20, end_bit = 21)]],
    'UserPhysicalPages' : [ 0x0, ['BitField', dict(start_bit = 21, end_bit = 22)]],
    'NoChange' : [ 0x0, ['BitField', dict(start_bit = 22, end_bit = 23)]],
    'WriteWatch' : [ 0x0, ['BitField', dict(start_bit = 23, end_bit = 24)]],
    'Protection' : [ 0x0, ['BitField', dict(start_bit = 24, end_bit = 29)]],
    'LargePages' : [ 0x0, ['BitField', dict(start_bit = 29, end_bit = 30)]],
    'MemCommit' : [ 0x0, ['BitField', dict(start_bit = 30, end_bit = 31)]],    
    'PrivateMemory' : [ 0x0, ['BitField', dict(start_bit = 31, end_bit = 32)]],
} ],

And then look at the same structure from the new win7 profile:

  '_MMVAD_FLAGS' : [ 0x4, {
    'CommitCharge' : [ 0x0, ['BitField', dict(start_bit = 0, end_bit = 19)]],
    'NoChange' : [ 0x0, ['BitField', dict(start_bit = 19, end_bit = 20)]],
    'VadType' : [ 0x0, ['BitField', dict(start_bit = 20, end_bit = 23)]],
    'MemCommit' : [ 0x0, ['BitField', dict(start_bit = 23, end_bit = 24)]],
    'Protection' : [ 0x0, ['BitField', dict(start_bit = 24, end_bit = 29)]],
    'Spare' : [ 0x0, ['BitField', dict(start_bit = 29, end_bit = 31)]],
    'PrivateMemory' : [ 0x0, ['BitField', dict(start_bit = 31, end_bit = 32)]],
} ],

Since vadinfo prints CommitCharge and Protection explicitly, and 
CommitCharge/Protection exist at the same bit offset in both structures, we 
print the right values by coincidence. However, when vadinfo prints Vad.Flags, 
the Win7 output is incorrect since the real Win7 struct has different members 
(for example it will say WriteWatch when really it should be MemCommit)

I recently bumped this from medium to high. Now I'm going to bump it from high 
to critical ;-p

Original comment by michael.hale@gmail.com on 29 Oct 2011 at 9:49

GoogleCodeExporter commented 9 years ago 
Well,

Hopefully (as has been done with other profiles), we can just insert this 
change in the profile that's first affected, and all the other should import 
from there...

It should just be a case of override the overlays before it's provided to that 
one profile.  Any chance someone could mock up a patch around that idea, and we 
can investigate whether that works/if there's problems, please?

Original comment by mike.auty@gmail.com on 9 Jan 2012 at 9:05

GoogleCodeExporter commented 9 years ago 
Ill make a patch and send it around tonight.

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 9:25

GoogleCodeExporter commented 9 years ago 
Hey Ikelos, 

You probably showed me this before, but I have to ask again. In xp_sp2_x86.py I 
have the following:

xpsp2overlays['_MMVAD_SHORT'] = [ None, {
    # This is the location of the MMVAD type which controls how to parse the
    # node. It is located before the structure.
    'Tag': [-4 , ['String', dict(length = 4)]],
    # This is the _MMVAD_SHORT.u.VadFlags structure (_MMVAD_FLAGS)
    'Flags': [ None, ['Flags', {'bitmap': {
        'PhysicalMapping': 19, 
        'ImageMap': 20, 
        'UserPhysicalPages': 21, 
        'NoChange': 22, 
        'WriteWatch': 23, 
        'LargePages': 29,
        'MemCommit': 30, 
        'PrivateMemory': 31 }, 
      'maskmap' : { 
        'CommitCharge': [0, 19], 
        'Protection': [24, 5],  
    }}]],
}]

The XP SP3 profile inherits everything directly, as there are no changes. 
However, starting with Win2k3 SP1, there are some differences, so in 
win2k3_sp1_x86.py I have the following:

win2k3sp1x86overlays = copy.deepcopy(win2k3_sp0_x86.win2k3sp0x86overlays) 

win2k3sp1x86overlays['_MMVAD_SHORT'] = [ None, {
    # This is the location of the MMVAD type which controls how to parse the
    # node. It is located before the structure.
    'Tag': [-4 , ['String', dict(length = 4)]],
    # This is the _MMVAD_SHORT.u.VadFlags structure (_MMVAD_FLAGS)
    'Flags': [ None, ['Flags', {
        'bitmap': {
    'NoChange': 19, 
    'MemCommit': 23, 
    'Spare': 29, 
    'PrivateMemory': 31 }, 
        'maskmap' : { 
    'CommitCharge': [0, 19], 
    'VadType': [20, 3], 
    'Protection': [24, 5],  
    }}]],
}]

Notice the XPSP2 structure has a member named UserPhysicalPages, but Win2K3SP1 
does not. When I test with vadinfo I get some output that shows 
UserPhysicalPages is a flag:

$ python vol.py vadinfo -f Win2K3SP1x86-7d9079a4.vmem vadinfo 
--profile=Win2K3SP1x86

VAD node @81b4e358 Start 76f00000 End 76f07fff Tag Vad 
Flags: UserPhysicalPages
Commit Charge: 1 Protection: 7
ControlArea @81e57c10 Segment e168f1e8

I tried using the following code instead and it resulted in the same output:

win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'] = [ None, ['Flags', {
        'bitmap': {
    'NoChange': 19, 
    'MemCommit': 23, 
    'Spare': 29, 
    'PrivateMemory': 31 }, 
        'maskmap' : { 
    'CommitCharge': [0, 19], 
    'VadType': [20, 3], 
    'Protection': [24, 5],  
    }}]]

How can I make sure to delete/remove all members of the XPSP2 structure before 
adding the new Win2K3SP1 structure?

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 10:37

GoogleCodeExporter commented 9 years ago 
Hmmmm,

That's pretty odd.  The overlay should just be a dictionary, so overwriting the 
entry should remove any underlying stuff.  So a few things to check are the 
output of repr(win2k3sp1x86overlays['_MMVAD_SHORT']) and see that it's 
definitely been overwritten, failing that we'll have to do some more invasive 
debugging...

As you can see, the DTB signature's being overridden correctly, so it may be 
something about the way we use _MMVAD as a factory.

Original comment by mike.auty@gmail.com on 9 Jan 2012 at 10:48

GoogleCodeExporter commented 9 years ago 
Hey Ikelos, 

If I put the following statement in win2k3_sp1_x86.py:

+ print repr(win2k3sp1x86overlays['_MMVAD_SHORT'])

win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset
win2k3sp1x86overlays['_CONTROL_AREA'][1]['Flags'][0] = lambda x: x.u.obj_offset

It prints: 

[None, {'Tag': [-4, ['String', {'length': 4}]], 'Flags': [None, ['Flags', 
{'maskmap': {'VadType': [20, 3], 'Protection': [24, 5], 'CommitCharge': [0, 
19]}, 'bitmap': {'NoChange': 19, 'PrivateMemory': 31, 'Spare': 29, 'MemCommit': 
23}}]]}]

So that looks OK right? But if I put a similar statement in vadinfo.py:

 outfd.write("Flags: {0}\n".format(vad.Flags))
 + print repr(vad.obj_vm.profile.overlay['_MMVAD_SHORT'])
 outfd.write("Commit Charge: {0} Protection: {1:x}\n".format(
     vad.Flags.CommitCharge,
     vad.Flags.Protection >> 24))

Those lines look this this:

[None, {'Tag': [-4, ['String', {'length': 4}]], 'Flags': [<function <lambda> at 
0x10064aa28>, ['Flags', {'maskmap': {'VadType': [20, 3], 'Protection': [24, 5], 
'CommitCharge': [0, 19]}, 'bitmap': {'NoChange': 19, 'PrivateMemory': 31, 
'Spare': 29, 'MemCommit': 23}}]]}]

So there are two 'Flags' members, one is a function. Perhaps I did something 
wrong or is that happening because of the way we use lambda here:

win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 11:05

GoogleCodeExporter commented 9 years ago 
I'm not seeing the following line in 
http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overl
ays/windows/win2k3_sp1_x86.py ?

win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset

It could well be that it's being overwritten but that should only be applied 
from u, you'd have to check the overlay for that to see what's going on there?  
I don't really have time to look further into it tonight, but I may be able to 
do some diagnostics tomorrow.  Any chance you could send a diff from the main 
trunk (not any of the branches) so I can run some tests myself please?

Sorry for the difficulty in getting this right, it should be simpler than 
this...

Original comment by mike.auty@gmail.com on 9 Jan 2012 at 11:19

GoogleCodeExporter commented 9 years ago 
Bit more info. So yeah if I print from win2k3_sp1_x86.py:

print repr(win2k3sp1x86overlays['_MMVAD_SHORT'])
win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset
print repr(win2k3sp1x86overlays['_MMVAD_SHORT'])

Then it shows:

[None, {'Tag': [-4, ['String', {'length': 4}]], 'Flags': [None, ['Flags', 
{'maskmap': {'VadType': [20, 3], 'Protection': [24, 5], 'CommitCharge': [0, 
19]}, 'bitmap': {'NoChange': 19, 'PrivateMemory': 31, 'Spare': 29, 'MemCommit': 
23}}]]}]
[None, {'Tag': [-4, ['String', {'length': 4}]], 'Flags': [<function <lambda> at 
0x10064aa28>, ['Flags', {'maskmap': {'VadType': [20, 3], 'Protection': [24, 5], 
'CommitCharge': [0, 19]}, 'bitmap': {'NoChange': 19, 'PrivateMemory': 31, 
'Spare': 29, 'MemCommit': 23}}]]}]

If I comment out the line like this:

#win2k3sp1x86overlays['_MMVAD_SHORT'][1]['Flags'][0] = lambda x: x.u.obj_offset

Then the plugin prints:

WARNING : volatility.obj      : Flags has no offset in object _MMVAD_SHORT. 
Check that vtypes has a concrete definition for it.

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 11:19

GoogleCodeExporter commented 9 years ago 
> I'm not seeing the following line 

Yeah, its just in a local copy I'm using to test. Sure I'll send you a diff 
before I sleep tonight...maybe I'll get it working with some trial and error ;-)

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 11:21

GoogleCodeExporter commented 9 years ago 
Oh nm it doesn't have two 'Flags' members. It looked like it at first (though 
being a dictionary its not even possible)

'Flags': [<function <lambda> at 0x10064aa28>, ['Flags', {'maskmap':

So the function should be ok, its just the lambda for the offset. That being 
said, I don't know how UserPhysicalPages is showing up when vad.Flags is 
printed since its not in the above bitmap.

Original comment by michael.hale@gmail.com on 9 Jan 2012 at 11:28

GoogleCodeExporter commented 9 years ago 
So, the offset shouldn't be the problem, the object should still be created 
with the members specified afterwards.  (You just said the same thing while I 
was typing).  5:)

It might be worth checking profile.typeDict['_MMVAD_SHORT'] to see if it has 
some issue with it.  The typeDict should be overridden by the overlay, and then 
when obj.Object() is called (in __new__ most likely), it'll try and look it up 
in the type (compiled) area, and if that fails, grab it directly from 
profile.object_classes.  I severely doubt that's happening, otherwise we'd be 
unlikely to see members, but it means it must be getting them from somewhere 
else.

It might be good to check _MMVAD_LONG as well (in case it's somehow falling 
through to that), and also verify that the Flags object's members are correct.  
I've really got to get some sleep now, but good luck for now, and I'll pick it 
up again in the morning...  5:)

Original comment by mike.auty@gmail.com on 9 Jan 2012 at 11:32

GoogleCodeExporter commented 9 years ago 
Hey Mike, 

> It might be good to check _MMVAD_LONG as well (in case it's somehow falling 
through to that)

I think that's what was happening. In fact it was a combination of things 
involving that:

* I didn't have an overlay for _MMVAD_LONG in win2k3_sp1_x86.py (the 
_MMVAD_LONG itself didn't change between OS)

* However, as you can see, _MMVAD_LONG has the Flags from _MMVAD_SHORT: 
http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overl
ays/windows/xp_sp2_x86.py#251

* The VADs I was seeing with UserPhysicalPages were _MMVAD_LONGs

So anyway, you should have an email with a patch using an entirely different 
technique suggested by Scudette which I think should fix things nicely.

Original comment by michael.hale@gmail.com on 10 Jan 2012 at 3:05

GoogleCodeExporter commented 9 years ago 
This issue was closed by revision r1238.

Original comment by michael.hale@gmail.com on 11 Jan 2012 at 2:42