raw2dmp - location to file conversion

[GoogleCodeExporter]

Hey Guys, 

Thanks again for the OMFW!

I think I found a bug with raw2dmp.

Bug seems to be related to a "location to file" conversion (don't know where is 
it in the code).

The location file:/// is not completly removed to get the file name (only 
file:// is removed). 

This doesn't works:

C:\Volatility\Volatility2.2>python vol.py -f 
C:/volatility/Volatility2.2/mem.vmem raw2dmp -O mem.dmp
Volatile Systems Volatility Framework 2.2
Writing data (5.00 MB chunks): |ERROR   : volatility.plugins.imagecopy: 
Unexpected error ([Error 123] The filename, dir
volume label syntax is incorrect: '/C:/volatility/Volatility2.2/mem.vmem') 
during copy, recorded data up to offset 0

This works:

C:\Volatility\Volatility2.2>python vol.py -l 
file://C:/volatility/Volatility2.2/mem.vmem raw2dmp -O mem.dmp
Volatile Systems Volatility Framework 2.2
Writing data (5.00 MB chunks): 
|...............................................................................
.......................
................................................................................
.......................|

Sebastien

Original issue reported on code.google.com by sebastie...@gmail.com on 3 Oct 2012 at 7:40

[GoogleCodeExporter]

Hi Sebastien,

I'm having a hard time recreating the problem you're experiencing?  Volatility 
is relatively forgiving and I've been able to raw2dump files using slashes in 
either direction, and locations with both two and three slashes. 

The file-to-location conversion (since volatility always uses locations at the 
lowest level) is stored in volatility/plugins/fileparam.py [1], and has special 
handling to ensure that file locations contain the correct number of slashes.

Could you please therefore double check a few things?  Firstly could you try:

-f C:\volatility\Volatility2.2\mem.vmem

then could you also try:

-l file:///C:/volatility/Volatility2.2/mem.vmem

Also could you please report which version of python you're using?  Volatility 
requires python 2.6 or python 2.7.  Hopefully that will help us figure out 
what's causing the problem...

[1] 
http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/filep
aram.py#39

Original comment by mike.auty@gmail.com on 3 Oct 2012 at 11:11

[GoogleCodeExporter]

Hi Mike,

Thanks for your help. 

I have try the conversion on 3 differents computer and they all failed (two 
Win7x64 SP1 and one XP64 SP2).

The version of python i'm using is:

C:\Volatility\Volatility2.2>python
ActivePython 2.7.2.5 (ActiveState Software Inc.) based on
Python 2.7.2 (default, Jun 24 2011, 12:21:10) [MSC v.1500 32 bit (Intel)] on 
win32

Here's the results of the new tests:

C:\Volatility\Volatility2.2>python vol.py -f 
C:\volatility\Volatility2.2\mem.vmem raw2dmp -O mem2.dmp
Volatile Systems Volatility Framework 2.2
Writing data (5.00 MB chunks): |ERROR   : volatility.plugins.imagecopy: 
Unexpected error ([Error 123] The filename, directory name, or
volume label syntax is incorrect: '/C:/volatility/Volatility2.2/mem.vmem') 
during copy, recorded data up to offset 0

C:\Volatility\Volatility2.2>python vol.py -l 
file:///C:/volatility/Volatility2.2/mem.vmem raw2dmp -O mem2.dmp
Volatile Systems Volatility Framework 2.2
Writing data (5.00 MB chunks): |ERROR   : volatility.plugins.imagecopy: 
Unexpected error ([Error 123] The filename, directory name, or
volume label syntax is incorrect: '/C:/volatility/Volatility2.2/mem.vmem') 
during copy, recorded data up to offset 0

I don't know if it's related, but in standard.py [1], there's 

path = urllib.url2pathname(config.LOCATION[7:])

Maybe the problem come from there? For the location file:///, maybe we should 
use config.LOCATION[8:] ?

C:\Volatility\Volatility2.2>python vol.py -l 
file:///C:/volatility/Volatility2.2/mem.vmem volshell
Volatile Systems Volatility Framework 2.2
Current context: process System, pid=4, ppid=0 DTB=0x6d00020
Welcome to volshell! Current memory image is:
file:///C:/volatility/Volatility2.2/mem.vmem
To get help, type 'hh()'
>>> self._config.LOCATION[7:]
'/C:/volatility/Volatility2.2/mem.vmem'
>>> exit()

Regards, 

Sebastien

[1] 
http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/addrs
paces/standard.py

Original comment by sebastie...@gmail.com on 4 Oct 2012 at 12:35

[GoogleCodeExporter]

Hi Sebastien,

url2pathname is an operating system specific python function, and it explicitly 
doesn't take the scheme into account (ie, you're supposed to remove file:// but 
no more).  If we remove file:/// then absolute filepaths in unix will suddenly 
become relative.

Also, python for windows should handle this, and I've installed ActiveState 
python and been unable to recreate your issue.  Both normal python and 
ActiveState python return the following:

> import urllib
> urllib.url2pathname("/C:/any/path/file.name")
"C:\\any\\path\\file.name"

So neither should be leaving the additional slash as the start of the path.  
Could you please check that your python installations correctly handle this 
case?

Also, you mentioned that you tested this on three 64-bit machines, but the 
python version you appear to be using is 32-bit.  Please could you try a native 
64-bit version of python, or test the same version of python on a 32-bit system?

So far, no other windows users have reported a problem accessing any files, and 
none of our developers have been able to recreate the same error yet, which 
strongly suggests this is something unique to your situation.  Can you think of 
any other differences between a normal setup and the systems you are testing 
this on?

Original comment by mike.auty@gmail.com on 4 Oct 2012 at 1:44

[GoogleCodeExporter]

Hi Mike,

I have tried 64 bit python and normal 32 python and conversion still fail... 

C:\Volatility\Volatility2.2>python vol.py -f 
c:\Volatility\Volatility2.2\mem.vmem raw2dmp -O mem2.dmp
Volatile Systems Volatility Framework 2.2
Writing data (5.00 MB chunks): |ERROR   : volatility.plugins.imagecopy: 
Unexpected error ([Error 123] The filename, directory name, or
volume label syntax is incorrect: '/C:/Volatility/Volatility2.2/mem.vmem') 
during copy, recorded data up to offset 0

C:\Volatility\Volatility2.2>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> urllib.url2pathname("/C:/any/path/file.name")
'C:\\any\\path\\file.name'

One of the difference I can think of is all my systems are configured in french.

>>> import locale
>>> locale.getdefaultlocale()
('fr_CA', 'cp1252')

Sebastien

Original comment by sebastie...@gmail.com on 4 Oct 2012 at 2:56

[GoogleCodeExporter]

Hey Sebastien, 

I can reproduce what you're seeing, so it definitely seems like a bug (and not 
specific to your french system). It affects the standalone 2.2 executable 
too...for example:

C:\Users\Jake\Desktop>volatility-2.2_rc2.standalone.exe -f clampi.vmem pslist
Volatile Systems Volatility Framework 2.2_rc2
Offset(V)  Name                    PID   PPID   Thds     Hnds   Sess  Wow64 
Start                Exit
---------- -------------------- ------ ------ ------ -------- ------ ------ 
-------------------- --------------------
0x823c8830 System                    4      0     64      263 ------      0
0x8210f648 smss.exe                588      4      3       21 ------      0 
2008-09-18 05:32:54
0x82267b98 csrss.exe               660    588     12      330      0      0 
2008-09-18 05:32:56
.....

C:\Users\Jake\Desktop>volatility-2.2_rc2.standalone.exe -f clampi.vmem raw2dmp 
-O clampi.dmp
Volatile Systems Volatility Framework 2.2_rc2
Writing data (5.00 MB chunks): |ERROR   : volatility.debug    : Unexpected 
error ([Error 123] The filename, directory name, or volume label syntax is 
incorrect:
 '/C:/Users/Jake/Desktop/clampi.vmem') during copy, recorded data up to offset 0

C:\Users\Jake\Desktop>

Original comment by michael.hale@gmail.com on 4 Oct 2012 at 7:45

[GoogleCodeExporter]

Just FYI this isn't a problem on Linux or Mac...raw2dmp works fine on those 
platforms. 

Original comment by michael.hale@gmail.com on 4 Oct 2012 at 7:49

[GoogleCodeExporter]

Mike,

If it can help, here's the a part of the result when I trace volatility:

[...]
 --- modulename: addrspace, funcname: write
addrspace.py(189):         if not self._config.WRITE:
addrspace.py(191):         self.data = self.data[:addr] + data + self.data[addr 
+ len(data):]
addrspace.py(192):         return True
raw2dmp.py(104):         path = self._config.LOCATION[7:]
 --- modulename: conf, funcname: __getattr__
conf.py(387):         if self.opts == None:
conf.py(391):         try:
conf.py(392):             return super(ConfObject, self).__getattribute__(attr)
conf.py(393):         except AttributeError:
conf.py(394):             pass
conf.py(398):         try:
conf.py(399):             return self.readonly[attr.lower()]
conf.py(400):         except KeyError:
conf.py(401):             pass
conf.py(404):         try:
conf.py(405):             return self.opts[attr.lower()]
raw2dmp.py(105):         num_pages = os.path.getsize(path) / 0x1000
 --- modulename: genericpath, funcname: getsize
genericpath.py(49):     return os.stat(filename).st_size
imagecopy.py(72):         except TypeError:
imagecopy.py(74):         except BaseException, e:
imagecopy.py(75):             debug.error("Unexpected error ({1}) during copy, 
recorded data up to offset {0:0x}".format(progress, str(e)))
[...]

Thanks again for your help,

Sebastien

Original comment by sebastie...@gmail.com on 4 Oct 2012 at 8:11

[GoogleCodeExporter]

Thanks Sebastien,

That helps a lot!  So I'm afraid I had a quick look through raw2dmp and 
couldn't find anything that differentiated it from imagecopy (which it's based 
upon), and so was using imagecopy for all my tests.  Your trace immediately 
identifies it as a problem further down and specific to raw2dmp, so we'll get 
that fixed up for you soon.  Thanks for reporting it!  5:)

Original comment by mike.auty@gmail.com on 4 Oct 2012 at 8:18

[GoogleCodeExporter]

I just read the trace again I think I saw the bug:

raw2dmp.py(104): path = self._config.LOCATION[7:]

should be:

path = urllib.url2pathname(config.LOCATION[7:])

Thanks again!

Original comment by sebastie...@gmail.com on 4 Oct 2012 at 8:24

[GoogleCodeExporter]

Ok, could you please try updating subversion to r2641 and let me know if that 
fixes your problem?

Original comment by mike.auty@gmail.com on 4 Oct 2012 at 8:24

[GoogleCodeExporter]

Problem solved!

Thank you ;)

Original comment by sebastie...@gmail.com on 4 Oct 2012 at 8:27

[GoogleCodeExporter]

No problem, Sebastien.  5:)

@MHL, how do you want to handle this?  Is it worth rolling a 2.2.1, and if so 
should we give it a few weeks before we do?  Also, should we close this off, or 
tag it as Milestone 2.2.1?  What're your thoughts on post-release bug 
management?  5:)

Original comment by mike.auty@gmail.com on 4 Oct 2012 at 8:29

[GoogleCodeExporter]

Nice, glad to see it was an easy fix. Hmm IMO it's not worth rolling a 2.2.1 at 
this point. Let's give it a few weeks like you said and see if any other bugs 
show up (there may have been one just posted on Vol-Users) so we can do them 
all together. I would be okay with closing this issue now that we have a patch 
and verification. I'll send you an email (and include some other devs) 
separately to talk about post-release bugs...sounds like a needed topic. 

Original comment by michael.hale@gmail.com on 4 Oct 2012 at 9:39

[GoogleCodeExporter]

Marking as fixed then!  5:)

Original comment by mike.auty@gmail.com on 5 Oct 2012 at 7:01

• Changed state: Fixed