VMware VMSS file format support as a new address space - code submission

[GoogleCodeExporter]

in the past few weeks (well, i may have been postponing this more then a few 
weeks :) )
I wrote an extension to Volatility's Address space layouts for VMWare's 
VMSS/VMSN file format, used in newer VMWare products to store the virtual 
machine's state when suspending or taking a snapshot.

the code I came up with is split in two:
VMSSParser.py is in charge of parsing the VMSS files and providing a generic 
access to the file's internal data types. I used a different file because i'm 
unfamiliar with volatility's structure parsing code but mostly because i would 
like the VMSSParser to be independent and used outside Volatility.
vmware.py implements the VMWareSnapshotFile address space. it just reads the 
VMSS file using VMSSParser to extract the memory regions for the virtual 
machine and provide an interface to the VMSSParser's methods, and then reads 
the CR0 register of the first core and sets the DTB for upper address spaces to 
take it from there.

I've used these for some time now and can't find any additional bugs, but it's 
important to remember VMSSParser was implemented without documentation and thus 
might not work for some VMSS files.
I used it against the Volatility 2.0 tree as well as the Volatility 2.1 alpha 
tree (dated to May 24) and it seems to work without any glitches.
I tried to write it as commented as possible.

Original issue reported on code.google.com by hmi...@gmail.com on 4 Jul 2012 at 9:58

Attachments:

• VMSNParser.py
• vmware.py

[GoogleCodeExporter]

Thanks very much, these look awesome!!!

I'll try to dig up a vmss file to run some tests, and give you some feedback.  
It's probably a little too late to make it into 2.1, but hopefully they'll make 
their debut in 2.2!  5:)

Original comment by mike.auty@gmail.com on 4 Jul 2012 at 12:53

[GoogleCodeExporter]

Original comment by mike.auty@gmail.com on 4 Jul 2012 at 12:53

• Added labels: Milestone-2.2.x

[GoogleCodeExporter]

sounds great, i hope they'd be helpful for someone.

i'd love any feedback i can get my hands on, especially from an official 
developer :)

Original comment by Nir...@gmail.com on 4 Jul 2012 at 12:59

[GoogleCodeExporter]

Yes very cool, thank you for the contribution. Coincidentally there was some 
discussion about parsing vmss files with volatility earlier this week: 
http://lists.volatilesystems.com/pipermail/vol-users/2012-July/000470.html

Original comment by michael.hale@gmail.com on 4 Jul 2012 at 2:29

[GoogleCodeExporter]

It looks to me like the address space is not implementing zread()
properly (or at all). Can you please make sure that you are
implementing zread() in such a way that when you read outside a valid
or mapped region you will receive a null padded buffer rather than
None?

Some more comments about the address space VMWareSnapshotFile:
   - Please do not use inner classes. There is no need to have a class
defined in such a way - just place the class at the module level. Also be 
consistent regarding class names (prefer CamelCase for classes).

   - Minor style issues - long lines should be wrapped at 80 chars,
commented out lines should be removed.

   - Do no use double underscore member variable names (they mean
something specific (non inheritable) e.g. self.__hasseek). Since the seek/read 
interface is totally different to the read(offset, length) interface I would 
not try to support both in the same class. I would just have 2 separate classes 
(one can inherit from the other).

   - It would also be nicer if we used the volatility object system
rather than struct module directly for parsing these things - it would
make the file formats more readable and simplify the code a lot. I can help you 
with that if you like to send me the documentation of the file format, or we 
can refactor it later.

Original comment by scude...@gmail.com on 6 Jul 2012 at 10:44

[GoogleCodeExporter]

Original comment by michael.hale@gmail.com on 26 Aug 2012 at 5:34

• Added labels: Milestone-2.3.x
• Removed labels: Milestone-2.2.x

[GoogleCodeExporter]

I opened a google code project for the vmsn parser, you can find it at 
http://code.google.com/p/vmsnparser/
hopefully sometime soon i'll start writing the actual format specs in the wiki 
but i'll have to make my notes more readable first :-)

I've also made a few fix-ups in the code, mostly to the volatility interface.
it should now support all the required methods according to the interface 
changes.
to do that i used two standard ASes from scudette's code branch. Mike Auty told 
me on IRC that including them won't be a problem.
These ASes are: RunBasedAddressSpace, PagedReader from 
branches/scudette/volatility/addrspace.py
to be able to use these you need to copy these classes from scudette's branch 
to the matching file in working copy.
I got to test it a bit using the single VMSN file i had on my laptop, but i can 
only test it properly at work (which is actually in a few hours :) )

As for scudette's comments:
 - Inner class was removed, regions are now implemented as they should in the runs member inherited from RunBasedAddressSpace.

 - haven't done this yet, but I also have other small code changes to make (most notably replace "BLAH" with actual error messages for as_assert()s i never got)

 - removed. i now directly use the file handler from the base AS (I assume it's a FileAddressSpace) but actually i'm not sure that's the best way to go...

 - As i said before, i'd like the file parser to be used outside the volatility framework and I don't want to have to maintain both code pieces. I don't see any way to make that possible using the volatility object system. I believe there won't be many changes once the parser is more mature, so we could probably work something out later.

We might want to move this discussion to the vmsnparser project, but I don't 
really mind continuing it here if you guys prefer.

 - Nir

Original comment by Nir...@gmail.com on 19 Sep 2012 at 12:02

[GoogleCodeExporter]

Hey Nir, 

Thanks a bunch. I starred your project ;-) Also, this is on the roadmap for 2.3 
(next volatility version) so we may be reaching out to you for some questions 
or testing as we work to implement your parser. Talk to you soon!

Original comment by michael.hale@gmail.com on 8 Oct 2012 at 10:03

[GoogleCodeExporter]

Hey Nir, 

Your VMware address space has made it into the volatility core, in time for the 
2.3 release: 

http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/addrs
paces/vmware.py

So look out for proper credits in the release notes, wiki, blog, etc. Also, 
everything below the line "FIXME: everything below can be abstract - shared 
with virtualbox" will end up being removed before the release. 

Thanks again, great work!

Original comment by michael.hale@gmail.com on 1 Feb 2013 at 4:59

• Changed state: Done

[GoogleCodeExporter]

Cool!

I already noticed the FIXME and was wondering how long it's gonna last there :P

please let me know if there's anything i can further assist with in the future.
bugs/new versions etc...

I hope it'll be put to some good use :)

Original comment by Nir...@gmail.com on 1 Feb 2013 at 6:35

[GoogleCodeExporter]

It looks good btw, waiting until i can try it for the first time :)

Original comment by Nir...@gmail.com on 1 Feb 2013 at 6:38