fatal exception acquiring an address space (vaddump, memdump, etc)

[GoogleCodeExporter]

Hey guys, 

This issue was reported to me today...wanted to get your opinion. Here's what I 
know: 

XP SP3 
2GB memory image
Copy of the stack dump: http://pastie.org/private/yv4jobxnqopfqllogh0eda

Looks like an issue acquiring the process address space for pid 1988. 

$ python /usr/local/src/Volatility/vol.py -f memdump.bin pslist -p 1988
Volatile Systems Volatility Framework 2.1_alpha Offset(V)  Name                 
PID    PPID   Thds   Hnds   Time 
---------- -------------------- ------ ------ ------ ------ ------------------- 
0x878a81d0 searchfilterhos        1988   2860      1    194 2011-07-25 17:05:15 

So the process is active, has threads, and the handle table is in tact. 

$ python /usr/local/src/Volatility/vol.py -f memdump.bin dlllist on -p 1988
Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
searchfilterhos pid:   1988
Unable to read PEB for task.

The memory image itself is sensitive so we can't have a copy for testing, but 
I've CC'd the reporting user (Brett) and he should be able to paste output to 
help troubleshoot or try other commands, etc. 

Original issue reported on code.google.com by michael.hale@gmail.com on 4 Oct 2011 at 7:29

[GoogleCodeExporter]

Hmmm, very odd.  Looks from the trace (copied below) as if it's an issue with 
the process's dtb pointing to a block that includes a reference to a space 
outside the raw disk.

Without a reproducable image, it's difficult to tell whether it's the DTB 
crossing page boundaries, or whether it's valid but corrupted memory that's 
trying to be read...

Pid:   1988
Traceback (most recent call last):
  File "/usr/local/src/Volatility/vol.py", line 135, in <module>
    main()
  File "/usr/local/src/Volatility/vol.py", line 126, in main
    command.execute()
  File "/usr/local/src/Volatility/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/usr/local/src/Volatility/volatility/plugins/vadinfo.py", line 176, in render_text
    task_space = task.get_process_address_space()
  File "/usr/local/src/Volatility/volatility/plugins/overlays/windows/windows.py", line 197, in get_process_address_space
    process_as = self.obj_vm.__class__(self.obj_vm.base, self.obj_vm.get_config(), dtb = directory_table_base)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 89, in __init__
    self.as_assert(getattr(volmag, self.checkname).v(), "Failed valid Address Space check")
  File "/usr/local/src/Volatility/volatility/obj.py", line 801, in v
    return self.get_best_suggestion()
  File "/usr/local/src/Volatility/volatility/obj.py", line 827, in get_best_suggestion
    for val in self.get_suggestions():
  File "/usr/local/src/Volatility/volatility/obj.py", line 819, in get_suggestions
    for x in self.generate_suggestions():
  File "/usr/local/src/Volatility/volatility/plugins/overlays/windows/windows.py", line 521, in generate_suggestions
    if (self.obj_vm.vtop(pde_base) == pd):
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 446, in vtop
    pte = self.get_pte(vaddr, pde)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 413, in get_pte
    return self._read_long_long_phys(pte_addr)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 458, in _read_long_long_phys
    string = self.base.read(addr, 8)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/standard.py", line 97, in read
    self.fhandle.seek(addr)
IOError: [Errno 22] Invalid argument

Original comment by mike.auty@gmail.com on 4 Oct 2011 at 7:38

[GoogleCodeExporter]

The output of running vadinfo:

$ python /usr/local/src/Volatility/vol.py -f memdump.bin vadinfo -p 1988
Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
Pid:   1988
Traceback (most recent call last):
  File "/usr/local/src/Volatility/vol.py", line 135, in <module>
    main()
  File "/usr/local/src/Volatility/vol.py", line 126, in main
    command.execute()
  File "/usr/local/src/Volatility/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/usr/local/src/Volatility/volatility/plugins/vadinfo.py", line 42, in render_text
    for vad in task.VadRoot.traverse():
  File "/usr/local/src/Volatility/volatility/obj.py", line 572, in __getattribute__
    result = self.dereference()
  File "/usr/local/src/Volatility/volatility/obj.py", line 548, in dereference
    name = self.obj_name)
  File "/usr/local/src/Volatility/volatility/plugins/overlays/windows/windows.py", line 363, in __new__
    vm = eprocess.get_process_address_space()
  File "/usr/local/src/Volatility/volatility/plugins/overlays/windows/windows.py", line 197, in get_process_address_space
    process_as = self.obj_vm.__class__(self.obj_vm.base, self.obj_vm.get_config(), dtb = directory_table_base)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 89, in __init__
    self.as_assert(getattr(volmag, self.checkname).v(), "Failed valid Address Space check")
  File "/usr/local/src/Volatility/volatility/obj.py", line 801, in v
    return self.get_best_suggestion()
  File "/usr/local/src/Volatility/volatility/obj.py", line 827, in get_best_suggestion
    for val in self.get_suggestions():
  File "/usr/local/src/Volatility/volatility/obj.py", line 819, in get_suggestions
    for x in self.generate_suggestions():
  File "/usr/local/src/Volatility/volatility/plugins/overlays/windows/windows.py", line 521, in generate_suggestions
    if (self.obj_vm.vtop(pde_base) == pd):
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 446, in vtop
    pte = self.get_pte(vaddr, pde)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 413, in get_pte
    return self._read_long_long_phys(pte_addr)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/intel.py", line 458, in _read_long_long_phys
    string = self.base.read(addr, 8)
  File "/usr/local/src/Volatility/volatility/plugins/addrspaces/standard.py", line 97, in read
    self.fhandle.seek(addr)

Original comment by Bret...@gmail.com on 4 Oct 2011 at 8:35

[GoogleCodeExporter]

Yep, that's exactly the same backtrace, so it's happening as it's trying to 
instantiate the _EPROCESS address space.  It checks that the DTB works by 
running a couple of tests.  The test in particular is the first one we run that 
looks for referential elements within the dtb...

http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overl
ays/windows/windows.py#521

Unfortunately that seems to be a pretty good test so far, which suggests that 
the dtb pulled from the eprocess is broken...

Original comment by mike.auty@gmail.com on 4 Oct 2011 at 8:40

[GoogleCodeExporter]

Another bit of into - the acquisition was made with fdpro.

So even if we are dealing with a bad acquisition (or a good acquisition but for 
some reason the translation tables are in a funny state at the time), we should 
still catch this exception so that vaddump moves onto other processes instead 
of halting completely). 

Original comment by michael.hale@gmail.com on 4 Oct 2011 at 8:40

[GoogleCodeExporter]

You're quite right, although I would like to get to the bottom of the issue 
itself (check whether it's a corrupt translation table, or a bug in 
volatility's test).

Either way, here's a patch to chew over for a little while, and if it tastes 
good then I'll apply it to trunk...  5:)

Original comment by mike.auty@gmail.com on 4 Oct 2011 at 8:47

Attachments:

• volatility-issue-146.patch

[GoogleCodeExporter]

I would suggest putting the IOError exception around vtop() and just return 
False. Its likely this kind of error will occur again in future.

The second question I have is the utility of this AS test. It seems to me more 
reliable to use the profile itself to tell the difference between the IA32 and 
IA64 AS's and the DTB scan tells you pretty reliably where in the voting level 
the intel AS should come. Should we really reject an image just because it 
looks slightly wrong?

That tests looks very windows specific and one would wonder if it can 
generalize to other windows versions let alone operating systems.

Original comment by scude...@gmail.com on 4 Oct 2011 at 9:18

[GoogleCodeExporter]

We probably don't want vm.vtop(a) == vm.vtop(b) being true if both a and b are 
outsite the address space and therefore False.  Is there another value whose 
__equals__ always returns False?

The reason for the test is not to determine whether the profile is valid or 
not, but to rule out bogus finds by the DTB signature engine.  The DTB 
signature finder is more of a problem than the check, although both can be 
overridden by the profile (meaning that it already *is* the profile which 
decides whether the AS is valid or not).

Currently the only DTB finder we have relies on an Idle process existing, and 
that's in basic.py not windows.py.  I haven't checked attc's code, but I don't 
think he's found a solution to it, he may just rely on grabbing it out of 
System.map or somewhere similar.  If you can come up with a good OS-agnostic 
scan for finding DTBs in arbitrary images, that'd be awesome!  5:)

Original comment by mike.auty@gmail.com on 4 Oct 2011 at 9:32

[GoogleCodeExporter]

Indeed, returning a NoneObject in case of an error during the vtop translation 
is the right thing to do :-)

Currently the DTB signature scanner looks for an "Idle" process. This is a very 
strong signature and can be made much stronger by using some of the self 
referential lists, pool tags and other _EPROCESS attributes. It is also very 
windows specific.

The DTB scanner is implemented as a volatility magic which is overlayed 
depending on the profile. So the linux one is different from the windows one 
(and infact does not need any extra DTB validation because linux has a simple 
constant relation between VAS and PAS in kernel mode). It does not make sense 
to me to have two sanity checks which are not related (one in the AS and one in 
the DTB scanner).

Original comment by scude...@gmail.com on 5 Oct 2011 at 12:52

[GoogleCodeExporter]

Hey guys, just an update for you. Thanks to Brett for patience and testing. 

First we converted the memory dump to a crash dump and loaded it in windbg. 
Then switched contexts to the offending process (pid 1988): 

kd> .process /p /r 0x878a81d0

Here's the output of !peb for that process:

Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE )

Followup: MachineOwner
---------

0: kd> .process /p /r 0x878a81d0
Implicit process is now 878a81d0
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details
0: kd> !peb
PEB at 7ffdf000
error 1 InitTypeRead( nt!_PEB at 7ffdf000)...

Then we tried to just convert the virtual address to physical, expecting windbg 
to throw errors if the DTB does really contain garbage:

0: kd> !vtop 0 7ffdf000
Pdi 3ff Pti 1df
PageDirectory Entry 1023 not valid, try another process

So I think this confirms that there's no issue with Volatility per se (other 
than it could handle the exception more gracefully). 

Original comment by michael.hale@gmail.com on 6 Oct 2011 at 3:30

[GoogleCodeExporter]

Hmmmm, be interesting to know how it determined the PageDirectory entry wasn't 
valid?  Be interesting to know what to do in that circumstance (presumably 
throw an exception, but I guess a more explanitory one)...

Original comment by mike.auty@gmail.com on 6 Oct 2011 at 5:59

[GoogleCodeExporter]

Anyone willing to write a verbose vtop() like function for volshell? Something 
that would output enough information on the page directory and PTEs w/ flags? 
Then Brett could just run the volshell command and gather enough info to paste 
back and help determine exactly what is invalid. 

Original comment by michael.hale@gmail.com on 28 Oct 2011 at 3:25

[GoogleCodeExporter]

Since we've determined that the address wasn't valid, I'm setting this as a low 
priority.  If you disagree, let me know and we can bump it back up...

Original comment by mike.auty@gmail.com on 12 Feb 2012 at 8:35

• Added labels: Priority-Low
• Removed labels: Priority-High

[GoogleCodeExporter]

Hey guys, its been 6 months and the issue hasn't been seen on any other dumps 
(not to mention Windbg agrees the page is invalid). I'm not sure we'll ever 
figure out *why* the page is invalid if we cant reproduce it again. We fixed 
the fatal exception, so IMO we can close this and reference/re-open in the 
future if needed. 

Original comment by michael.hale@gmail.com on 9 Aug 2012 at 4:34

• Changed state: Fixed