false positive with psscan [bad bounce from physical proc to virtual?]

[GoogleCodeExporter]

we all know scanners can yield false positives, but i'm logging it here in case 
someone wants to take a look at how/why this particular case bypasses the 
constraints (and possible suggest a way to fix it).

memory image: 
https://domex.nps.edu/corp/scenarios/2009-m57/ram/jo-2009-12-11.mddramimage.zip

$ python vol.py -f ~/Downloads/jo-2009-12-11.mddramimage psscan
0x05cc7da0 avgwdsvc.exe       1788    976 0x26ec5000 2009-12-11 19:05:36        

0x05cd7500 avgrsx.exe         1564    932 0x21ce8000 2009-12-11 19:05:19        

0x05cd9408 svchost.exe        1568    976 0x21ce1000 2009-12-11 19:05:19        

0x05d1b890                  544240201 2242963456 0x8639b8e0                     

0x05d21338 csrss.exe           908    812 0x1ebe2000 2009-12-11 19:05:17        

0x05d34da0 avgcsrvx.exe       1828   1564 0x22d37000 2009-12-11 19:05:21

Original issue reported on code.google.com by michael.hale@gmail.com on 12 Oct 2011 at 7:24

[GoogleCodeExporter]

Hmm, little more info...

If you try procexedump on a process that has exited (this Peb isn't available 
and address space is gone), you'll get something like:

Error: PEB not memory resident for process [-]

However if you try procexedump on the EPROCESS above at 0x05d1b890, it will 
give you this:

************************************************************************
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/Users/Mike/Desktop/volatility_21alpha/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/Users/Mike/Desktop/volatility_21alpha/volatility/plugins/procdump.py", line 50, in render_text
    task_space = task.get_process_address_space()
  File "/Users/Mike/Desktop/volatility_21alpha/volatility/plugins/overlays/windows/windows.py", line 197, in get_process_address_space
    process_as = self.obj_vm.__class__(self.obj_vm.base, self.obj_vm.get_config(), dtb = directory_table_base)
  File "/Users/Mike/Desktop/volatility_21alpha/volatility/plugins/addrspaces/intel.py", line 85, in __init__
    self._cache_values()
  File "/Users/Mike/Desktop/volatility_21alpha/volatility/plugins/addrspaces/intel.py", line 119, in _cache_values
    self.pde_cache = struct.unpack('<' + 'I' * 0x400, buf)
struct.error: unpack requires a string argument of length 4096

So the entry is able to be found by psscan, it "passes" the 
virtual_process_from_physical_offset() function (although that's not really a 
constraint/check) but crashes out when trying to go further. 

Again I didn't actually expect procexedump to work, but we should fail 
gracefully in these cases. 

Original comment by michael.hale@gmail.com on 12 Oct 2011 at 7:45

[GoogleCodeExporter]

Hey guys, users reported two similar errors to Gleeda today. I'm going to 
discuss them in a few comments here. 

The first one was using the strings command with -S option (use psscan). Full 
command:

python vol.py -f ../memory_dumps/image.bin --profile=WinXPSP3x86 strings -s 
../memory_dumps/strings_output/image_str.txt 
--output-file=../memory_dumps/strings_output/image_vol.txt -S

Stack dump:

Volatile Systems Volatility Framework 2.1_alpha
Traceback (most recent call last):
File "vol.py", line 135, in <module>
main()
File "vol.py", line 126, in main
command.execute()
File "/Volatility/volatility/commands.py", line 101, in execute
func(outfd, data)
File "/Volatility/volatility/plugins/strings.py", line 95, in render_text
reverse_map = self.get_reverse_map(addr_space, tasks, verbfd)
File "/Volatility/volatility/plugins/strings.py", line 141, in get_reverse_map
task_space = task.get_process_address_space()
File "/Volatility/volatility/plugins/overlays/windows/windows.py", line 197, in 
get_process_address_space
process_as = self.obj_vm.__class__(self.obj_vm.base, self.obj_vm.get_config(), 
dtb = directory_table_base)
File "/Volatility/volatility/plugins/addrspaces/intel.py", line 85, in _init_
self._cache_values()
File "/Volatility/volatility/plugins/addrspaces/intel.py", line 360, in 
_cache_values
self.pdpte_cache = struct.unpack('<' + 'Q' * 4, buf)
struct.error: unpack requires a string argument of length 32

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:09

• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Regarding comment #2 (user running strings with -S), he said there's no crash 
when omitting the -S flag. 

Please see the attached pslist and psscan output. At first we figured that 
since a crash occurs with -S but is OK when -S is not used, then psscan may be 
picking up a false positive. 

However, when you compare pslist and psscan, you see there are exactly 103 
processes shown by both commands. psscan doesn't find anything that pslist 
doesn't find. And nothing in pslist looks crazy, right?

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:15

• Changed title: false positive with psscan [bad bounce from physical proc to virtual?]

Attachments:

• pslist.txt
• psscan.txt

[GoogleCodeExporter]

So we asked ourselves what could be different between how strings analyzes 
processes found using -S as opposed to not using -S. The difference is use of 
the taskmods.DllList.virtual_process_from_physical_offset function. This 
function assumes that the process is alive and well (its process space is 
accessible). If the process has terminated and members are partially 
overwritten (or if there are simply 0 threads left) then the bounce-back from 
physical to virtual (which relies on dereferencing the ETHREAD) will fail. 
There are two ways to fail. We only cover one of them in the existing code. For 
example if ethread.ThreadsProcess is an invalid pointed (like 0x123) then we'll 
return obj.NoneObject. However, if ethread.ThreadsProcess is valid (i.e. points 
in non-paged kernel memory) then we assume its pointing to the EPROCESS. 

I wrote a small plugin for the reporting user to run. It looks like this:

class BadBounce(taskmods.DllList):

    def calculate(self):
        addr_space = utils.load_as(self._config)        

        for proc in filescan.PSScan(self._config).calculate():
            virtual_proc = self.virtual_process_from_physical_offset(addr_space, proc.obj_offset)
            yield proc, virtual_proc 

    def render_text(self, outfd, data):
        for proc, virtual_proc in data:

            if virtual_proc == None:
                space_status = 'Cannot bounce, handled OK'
                virtual_offset = 0
            else:
                virtual_offset = virtual_proc.obj_offset 
                try:
                    virtual_proc.get_process_address_space()
                except: 
                    space_status = 'Exception, bad bounce!'
                else:
                    space_status = 'OK'

            outfd.write("Offset(P): {0:#x} Offset(V): {1:#x} Space {2}\n".format(proc.obj_offset, virtual_offset, space_status)) 

Here's a snippet of the output:

Offset(P): 0x5b9fda0 Offset(V): 0x85d9fda0 Space OK
Offset(P): 0x5c7e020 Offset(V): 0x8a960ba0 Space OK
Offset(P): 0x5cbd020 Offset(V): 0x85f15974 Space Exception, bad bounce!
Offset(P): 0x5d0b5d8 Offset(V): 0x85f0b5d8 Space OK
Offset(P): 0x5d157f0 Offset(V): 0x860328c0 Space OK
Offset(P): 0x5d5ada0 Offset(V): 0x85f5ada0 Space OK
Offset(P): 0x5d5b020 Offset(V): 0x85f5b020 Space OK
Offset(P): 0x5d6f580 Offset(V): 0x85f6f580 Space OK
Offset(P): 0x5d77020 Offset(V): 0x85f77020 Space OK
Offset(P): 0x5d7ab78 Offset(V): 0x85f7ab78 Space OK
Offset(P): 0x5da6368 Offset(V): 0x85fa6368 Space OK
Offset(P): 0x5dbe728 Offset(V): 0x85fbe728 Space OK
Offset(P): 0x5e13990 Offset(V): 0x86013990 Space OK
Offset(P): 0x5e4e950 Offset(V): 0x8604e950 Space OK
Offset(P): 0x5e6f2b0 Offset(V): 0x8606f2b0 Space OK
Offset(P): 0x5e72020 Offset(V): 0x86072020 Space OK
Offset(P): 0x5e754d0 Offset(V): 0x860754d0 Space OK
Offset(P): 0x5e919d8 Offset(V): 0x860919d8 Space OK
Offset(P): 0x5e9d020 Offset(V): 0x0 Space Cannot bounce, handled OK
Offset(P): 0x5ec7590 Offset(V): 0x860c7590 Space OK
Offset(P): 0x5ecf500 Offset(V): 0x860cf500 Space OK

As you can see, the exception occurs on the eprocess at 0x5cbd020 found by 
psscan. The virtual_process_from_physical_offset function bounce-back says the 
virtual eprocess for 0x5cbd020 is 0x85f15974...which is clearly not correct (at 
least it doesn't follow the same pattern as the others). So based on the 
0x??bd020 part of 0x5cbd020, I *think* the bounce-back should have identified 
this process from pslist:

0x85ebd020 iexplore.exe           3992   6008      0 ------ 2011-11-01 18:06:13 

As you can see, it has 0 threads and ----- handles. 

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:29

[GoogleCodeExporter]

Sorry for the multiple comments....

Anyway, it sounds like we need a way to verify that the process returned by 
virtual_process_from_physical_offset is really a process and not just some 
pointer. One method is to return obj.NoneObject if flateproc.ActiveThreads == 0 
(before we dereference the flateproc.ThreadHeadList.Flink)

Another way, which is kinda sketchy (maybe ikelos or scudette can say if this 
will work) is to compare the last 4 digits or so of the virtual offset with the 
physical offset:

if physical_proc.obj_offset & 0xFFFF != virtual_proc.obj_offset & 0xFFFF:
    return obj.NoneObject("....")

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:35

[GoogleCodeExporter]

Here's the second report we received today. From a different user and different 
command. Here is the command:

python Volatility/vol.py --f forensic\ images/memory.img procexedump --dump-dir 
dumps/

Here is the stack dump:

Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
Error: PEB not memory resident for process [4]

-- cut for brevity

Dumping cmd.exe, pid:   4324 output: executable.4324.exe
************************************************************************
Error: PEB not memory resident for process [7356]
************************************************************************
Error: PEB not memory resident for process [6712]
************************************************************************
Error: PEB not memory resident for process [7700]
************************************************************************
Traceback (most recent call last):
  File "Volatility/vol.py", line 135, in <module>
    main()
  File "Volatility/vol.py", line 126, in main
    command.execute()
  File "/Users/lucien/Volatility/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/Users/lucien/Volatility/volatility/plugins/procdump.py", line 50, in render_text
    task_space = task.get_process_address_space()
  File "/Users/lucien/Volatility/volatility/plugins/overlays/windows/windows.py", line 197, in get_process_address_space
    process_as = self.obj_vm.__class__(self.obj_vm.base, self.obj_vm.get_config(), dtb = directory_table_base)
  File "/Users/lucien/Volatility/volatility/plugins/addrspaces/intel.py", line 85, in __init__
    self._cache_values()
  File "/Users/lucien/Volatility/volatility/plugins/addrspaces/intel.py", line 360, in _cache_values
    self.pdpte_cache = struct.unpack('<' + 'Q' * 4, buf)
struct.error: unpack requires a string argument of length 32

The thing that concerns me about this is that procexedump uses only processes 
in pslist and does not rely on the 
virtual_process_from_physical_offset...meaning by adding constraints for 
virtual_process_from_physical_offset we'll be fixing the first two issues (see 
comments #1 and #2) but not this one. 

So to fix this one, it seems like we need some sanity checks on the dtb or 
individual page entries parsed by get_process_address_space(). 

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:38

[GoogleCodeExporter]

I have to say, my bleary eyes think that your first suggestion is definitely 
the nicer one.  Theoretically ASLR won't ever alter the last 3 digits of the 
address, but I can't comment on the fourth digit.  I'd far sooner go with the 
pre-dereferencing check, the only question is, are there any situations where 
ActiveThreads != 0 but it's still a duff process?

Also, great work on the excellent analysis/diagnosis!  5:)

Original comment by mike.auty@gmail.com on 3 Nov 2011 at 10:40

[GoogleCodeExporter]

> are there any situations where ActiveThreads != 0 but it's still a duff 
process?

Good question. I wouldn't think so, but it would require some testing to say 
for sure. 

How about this third method though: 

if physical_proc.obj_offset != 
virtual_proc.obj_vm.vtop(virtual_proc.obj_offset):
    return obj.NoneObject(".....")

Wouldn't that tell us if the bounce worked properly, without checking active 
threads?

Btw, any thoughts on the stack dump in comment #6 that doesn't use 
get_process_address_space()

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:51

[GoogleCodeExporter]

> Btw, any thoughts on the stack dump in comment #6 that doesn't use 
get_process_address_space()

Oops I meant "that doesn't use virtual_process_from_physical_offset()"

Original comment by michael.hale@gmail.com on 3 Nov 2011 at 10:53

[GoogleCodeExporter]

Checking the obj_offset is the same as the vtop of the virtual obj_offset is 
good, but I wouldn't rely on it alone.

Hmmmm, best guess is that part of the Peb's valid, and the rest isn't, so the 
DTB isn't, but I'm so sleepy it might as well be a llama causing the problems.  
Something I'd have to look into further on the weekend I'm afraid.  Sorry...  
5:(

Original comment by mike.auty@gmail.com on 3 Nov 2011 at 10:55

[GoogleCodeExporter]

Im not sure any of this is unexpected? Is there more happening here than simply 
some process's dtb and page tables are unavailable (paged out/overwritten)? BTW 
are these users running with the transition patch enabled? - this might help in 
finding some more pages.

I think a better check for the validity of a process is to check the 
Flink/Blink of one (or more) of its many lists. e.g.

http://code.google.com/p/volatility/source/browse/branches/lin64-support/volatil
ity/plugins/filescan.py#450

Maybe we need to have a slightly better error reported through the intel AS.

Original comment by scude...@gmail.com on 3 Nov 2011 at 10:57

[GoogleCodeExporter]

One guy says: "both the 1.3 version , the 2.0 version and the 2.1 alpha version 
have the same issue."

The other says he's using r1119

Original comment by jamie.l...@gmail.com on 4 Nov 2011 at 2:45

[GoogleCodeExporter]

Sorry the first guy is having the issue in comment #6

The second one (r1119) is from comment #2

Original comment by jamie.l...@gmail.com on 4 Nov 2011 at 2:47

[GoogleCodeExporter]

Howdy, we have an update and a patch for this now. 

> Im not sure any of this is unexpected? Is there more happening here than 
simply some 
> process's dtb and page tables are unavailable (paged out/overwritten)?

Negative. If you look at the psscan output from the first comment, the 
problematic line is:

Offset                             Pid              Parent Pid      DTB
0x05d1b890                  544240201 2242963456 0x8639b8e0  

So the DTB is 0x8639b8e0, which is passed as the dtb parameter to instantiation 
of JKIA32PagedMemory in EPROCESS.get_process_address_space. In 
JKIA32PagedMemory.__init__ there is a call to self._cache_values() which does 
this:

buf = self.base.read(self.dtb, 0x1000)
if buf is None:
    self.cache = False
else:
    self.pde_cache = struct.unpack('<' + 'I' * 0x400, buf)

self.base in this case is FileAddressSpace and self.base.read is the Python 
File I/O read() function which never returns None. It returns an empty string 
like '' if the requested data/offset is available. 

So the first thing we must do to fix this issue is apply the following:

- if buf is None
+ if buf == ''

The second part of the patch involves adding the sanity check in 
DllList.virtual_process_from_physical_offset. As Ikelos said, this is a good 
check, but we shouldn't rely on it alone (which is why the first part of the 
patch is so important). Plus Scudette uses a similar sanity check in the 64 bit 
branch already. 

A full patch is attached, please give it a look. 

Note: I have yet to search for other calls to address_space.read that compare 
the result with None. 

Original comment by michael.hale@gmail.com on 14 Dec 2011 at 5:57

• Changed state: Started

Attachments:

• issue_154.patch

[GoogleCodeExporter]

Ok, the first part of the patch has been checked in as r1301.

Original comment by mike.auty@gmail.com on 23 Jan 2012 at 9:03

[GoogleCodeExporter]

Is the second part adding the safe bounce check described in comment #8?

Original comment by michael.hale@gmail.com on 24 Jan 2012 at 8:10

[GoogleCodeExporter]

Errr, it was ever so slightly more complex to read, and I was in a bit of a 
rush.  As far as I can tell (from comment 14):

"The second part of the patch involves adding the sanity check in 
DllList.virtual_process_from_physical_offset."

I've had chance to actually read it now, and it looks ok, feel free to commit 
(or let me know and I can do it)...  5:)

Original comment by mike.auty@gmail.com on 24 Jan 2012 at 8:35

[GoogleCodeExporter]

This issue was closed by revision r1310.

Original comment by michael.hale@gmail.com on 25 Jan 2012 at 3:18

• Changed state: Fixed