get_available_pages returns unavailable pages?

[GoogleCodeExporter]

I was testing memdump and noticed:

$ python vol.py -f memory_capture.dd --profile=Win7SP0x64 memdump -p 4 -D out
Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
Writing System [     4] to 4.dmp
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/Users/Michael/volatility_trunk_x64/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/Users/Michael/volatility_trunk_x64/volatility/plugins/taskmods.py", line 203, in render_text
    f.write(data)
TypeError: argument 1 must be convertible to a buffer, not NoneObject

Its not just the new x64 profiles because it happens on x86 also:

$ python vol.py -f win7vss.vmem --profile=Win7SP1x86 -D out/ memdump
Volatile Systems Volatility Framework 2.1_alpha
************************************************************************
Writing System [     4] to 4.dmp
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/Users/Michael/volatility_trunk_x64/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/Users/Michael/volatility_trunk_x64/volatility/plugins/taskmods.py", line 203, in render_text
    f.write(data)
TypeError: argument 1 must be convertible to a buffer, not NoneObject

I added some print statements before the f.write(data) line if data == None and 
here's what I saw (an x indicates data == None)

    0xf6fc4000c000
    0xf6fc4000d000
    0xf6fc4000e000
    0xf6fc4000f000
 x  0xf6fc40010000
 x  0xf6fc40011000
 x  0xf6fc40012000
 x  0xf6fc40013000
    0xf6fc40014000
    0xf6fc40015000
    0xf6fc40016000
    0xf6fc40017000
    0xf6fc40018000

Using volshell I can switch into the process context and dd 0xf6fc4000f000 just 
fine but 0xf6fc40010000 failed. 

After applying a patch 
(http://code.google.com/p/volatility/source/detail?r=1287) I got to 
thinking...these pages are returned by get_available_pages(). Shouldn't only 
memory-resident pages be returned by get_available_pages? 

Original issue reported on code.google.com by michael.hale@gmail.com on 20 Jan 2012 at 11:49

[GoogleCodeExporter]

Original comment by michael.hale@gmail.com on 20 Jan 2012 at 11:49

[GoogleCodeExporter]

They probably should.  Just to make sure it's that the pages aren't resident, 
could you please print out the .reason on any NoneObjects you come across (or 
bump up the -d until you get messages about it)...

Original comment by mike.auty@gmail.com on 21 Jan 2012 at 11:17

[GoogleCodeExporter]

Ok, so this was basically being masked before r1164, because we were returning 
'' when we failed to read a physical address correctly.  We now return 
NoneObjects.  I'm going to trace back and fine out if this has in fact always 
been a problem...

Original comment by mike.auty@gmail.com on 21 Jan 2012 at 1:39

[GoogleCodeExporter]

This appears to have been present in volatility-2.0, so I'm not sure we've ever 
had it working.

Just to give us something tangible to work with, on the publicly available 
xp-laptop-2005-07-04-1430 image, there's an available page (returned by 
get_available_pages) at 0xf6927000 that maps to a physical address of 
0xfd000000, which is way outside the bounds of the filesize (0x1ffda000).  I 
also have 0xffd00000 virtual mapping back to 0x1fff0000 physical, which is 
again outside the bounds, but this time by a much smaller amount...

Anyone who knows what they're doing care to investigate the pages and try to 
figure out what's different about them?

Original comment by mike.auty@gmail.com on 21 Jan 2012 at 2:01

[GoogleCodeExporter]

Ok, so here's an enhanced version of the pagechecking plugin from issue 214, 
and the output when applied to the two public NIST laptop images.  Weirdness 
abound...

Original comment by mike.auty@gmail.com on 19 Mar 2012 at 9:24

Attachments:

• xp-laptop-2005-06-25-invalid-pages.txt
• xp-laptop-2005-07-04-1430-invalid-pages.txt
• pagecheck.py

[GoogleCodeExporter]

Original comment by mike.auty@gmail.com on 19 Mar 2012 at 9:51

• Added labels: Milestone-2.1.x, Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

This looks to me like a page specified in the device io range. For example

http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b4
5/mem-mgmt.doc

explains Device-Bus-Relative Physical Addresses which are implemented in 
hardware to access hardware (probably video hardware in this case). Usually 
these addresses are put at the top of the physical range like here so they dont 
get in the way of ram.

There is a nice diagram that shows this here:
http://www.codinghorror.com/blog/2007/03/dude-wheres-my-4-gigabytes-of-ram.html

As far as the page table is concerned, this pte is valid and points to this 
area. Alas the memory capture did not capture this area (probably because it 
would crash the box to do this).

Original comment by scude...@gmail.com on 19 Mar 2012 at 10:40

[GoogleCodeExporter]

Ah, great news. Scudette is exactly right, I just pulled IO memory ranges from 
xp-laptop-2005-06-25 and xp-laptop-2005-07-04 and verified that the unavailable 
pages (yes, all 4000+ in both of Ikelos' log files) are in the reserved device 
memory ranges. 

So I would say our current handling of things is just right, and no changes are 
needed. Well, the only change needed was already applied in r1287 to just 
prevent memdump from crashing when address_space.read() fails on pages marked 
available. 

Lastly, Ikelos, I think we should throw pagecheck.py in contrib or something. 
Its come in handy troubleshooting at least two issues in the past few weeks. At 
the very least it could be useful to show the "holes" in memory due to devices. 

I'm going to close this issue and mark it as WontFix, because there's nothing 
that really needs fixing. I'll open a separate issue for discussions related to 
adding pagecheck.py just so we don't forget. 

Original comment by michael.hale@gmail.com on 3 Apr 2012 at 3:27

• Changed state: WontFix