Error reading with printkey plugin a registry value greater than 0x4000 bytes

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Create a REG_BINARY KEY with 0x4001 bytes
2. Use printkey to read that value
3. Volatility don't show the value. It use a lot of CPU.

What is the expected output? What do you see instead?
I expect the value but I see:
REG_BINARY    test            : (S)

What version of the product are you using? On what operating system?
Volatility 2.2 in Windows 7. 
This problem doesn't appears in Volatility 2.0.

Original issue reported on code.google.com by xyzt...@gmail.com on 8 Nov 2012 at 11:46

[GoogleCodeExporter]

I'm using as memfile the vmware .vmem file generated from a Windows XP SP3. If 
you need more information ask for it.

Regards.

Original comment by xyzt...@gmail.com on 8 Nov 2012 at 3:49

[GoogleCodeExporter]

Out of curiosity, does it work when you print out a known large data key, like 
so:

python vol.py [sample] printkey -K "ControlSet001\Control\Session 
Manager\AppCompatibility"

Original comment by jamie.l...@gmail.com on 8 Nov 2012 at 4:38

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi, 
With that subkey works good and it's a large key (0xD090 bytes). 

But then I created a key with the same values, the 0xD090 bytes in this subkey:
HKEY_CURRENT_USER\Software\Microsoft\Zizofy
and the error appears.
vol.py -f "xxxx.vmem" printkey -K "Software\Microsoft\Zizofy"
Volatile Systems Volatility Framework 2.2
Legend: (S) = Stable   (V) = Volatile

----------------------------
Registry: \Device\HarddiskVolume1\Documents and Settings\User\NTUSER.DAT
Key name: Zizofy (S)
Last updated: 2012-11-08 16:52:09

Subkeys:

Values:
REG_BINARY    Test       : (S)

Original comment by xyzt...@gmail.com on 8 Nov 2012 at 4:59

[GoogleCodeExporter]

This is strange.  So I must ask: how did you create the key?  

I have not yet been able to duplicate your problem.  I am about to try again on 
a Win7 machine.  

Here you can see output from a key that I created for testing:

http://pastebin.com/JyASzJh9

Original comment by jamie.l...@gmail.com on 8 Nov 2012 at 5:13

[GoogleCodeExporter]

Ok, finally I managed to duplicate this bug.  So I'm not exactly sure what the 
issue is yet, but here's what I know:

For experiments I created key values of size 0xd7550

1) The problem seems to manifest itself with large key values in NTUSER.DAT
2) I had no problems when I created large key values in SYSTEM
3) 2.0 handles large key values in both registries and 2.2 works only on the 
large key value in the SYSTEM registry and crashes with the one in NTUSER.DAT:

Traceback (most recent call last):
  File "./vol.py", line 186, in <module>
    main()
  File "./vol.py", line 177, in main
    command.execute()
  File "/volatility-2.2/volatility/commands.py", line 111, in execute
    func(outfd, data)
  File "/volatility-2.2/volatility/plugins/registry/printkey.py", line 113, in render_text
    dat = "\n" + "\n".join(["{0:#010x}  {1:<48}  {2}".format(o, h, ''.join(c)) for o, h, c in utils.Hexdump(dat)])
  File "/volatility-2.2/volatility/utils.py", line 70, in Hexdump
    for offset in xrange(0, len(data), width):
TypeError: object of type 'NoneType' has no len()

Stay tuned...

Original comment by jamie.l...@gmail.com on 8 Nov 2012 at 6:23

[GoogleCodeExporter]

Hi, 
have you discover the bug?

Thanks for your help, regards!

Original comment by xyzt...@gmail.com on 14 Nov 2012 at 12:14

[GoogleCodeExporter]

OK, the issue seems to be with some changes we made to rawreg.py in issue 169.  
So the problem is that without the patch in issue 169, we are unable to parse 
values that are of type _CM_BIG_DATA such as the 
AppCompatibility/AppCompatCache and rdyboost keys in the SYSTEM hive.  It 
seems, however, that extremely large registry values in NTUSER.DAT registries 
are handled differently than _CM_BIG_DATA.  

If you look at the `Signature` member of a _CM_BIG_DATA value in the SYSTEM 
hive you should see a value of 0x6264 (for naturally occurring keys/values like 
AppCompatibility as well as my manually created key and value that you can see 
in the pastebin above).  When I check the `Signature` for this value in an 
NTUSER.DAT hive, I had a different value of 0x8888.  Therefore, I think 
checking the signature for the known value may solve this problem.  

You can see a new run after applying this patch here: 
http://pastebin.com/Ayumm0CK I couldn't include all output, however, since it 
is too large and accidentally cut off the header when I copied and pasted, but 
was run using the latest code from svn r2794.

I've attached a patch, but it probably needs more testing before we absorb it.

In short, this is a valid bug and should be fixed by the next release.  Thanks 
for reporting it,  xyzthor!

Original comment by jamie.l...@gmail.com on 19 Nov 2012 at 7:38

Attachments:

• rawreg_patch.diff

[GoogleCodeExporter]

So the patch looks ok, but I'm just trying to get my head around the new 
signature (8888), since it looks very much like the first two bytes of the 
data, rather than a signature at all.  Were you able to recreate this, and if 
so could you please try different initial text (something like Lorem ipsum)?  
I'd like to know by just how much NTUSER.DAT differs from _CM_BIG_DATA/normal 
registry structure...

Original comment by mike.auty@gmail.com on 20 Nov 2012 at 9:22

[GoogleCodeExporter]

Mike: Yes you are right the "Signature" of 0x8888 is not a signature at all and 
is the registry key data.  This all makes sense because this particular value 
is not being stored as _CM_BIG_DATA. If you look in the documentation I 
provided in issue 169 (and if you google for more references), you will see 
that a valid signature for _CM_BIG_DATA is as I have described: 0x6264 

I tested this morning on a Windows 7 x86 machine to see if I could duplicate 
this behavior and I did.  I created large key values (with repeating data of 
0xDEADBEEF size 0x107cf) in the following registries at different locations to 
see how they would be handled:

SAM
SYSTEM
SOFTWARE
NTUSER.DAT

The SYSTEM and SOFTWARE registries stored the data as _CM_BIG_DATA, while the 
SAM and NTUSER.DAT registries stored the data as regular registry data.  So it 
appears that in certain registries large data is stored differently so we can't 
just assume that large key value data will always be stored as _CM_BIG_DATA and 
how it is stored may be dependent on the registry involved.  So far across two 
systems of different OSes I have seen the same behavior as I have just 
described here.

I can see one potential problem with this particular patch however: if the data 
in NTUSER.DAT begins with the signature for _CM_BIG_DATA.  In that case we 
might want to know the path of the registry at which we are looking and force a 
read of raw data.

Original comment by jamie.l...@gmail.com on 20 Nov 2012 at 3:21

[GoogleCodeExporter]

FYI the SAM and NTUSER.DAT are REGF version 1.3 and SYSTEM and SOFTWARE version 
1.5

According to [MSDN] the value data has a maximum size of the available memory 
in the latest format (1.5) and 1 MiB in the standard format (1.3). In the 
latest format (1.5) values larger than 16344 bytes are stored in multiple 
segments. Data about these segments is stored in the data block key. These 
large values are also referred to as long values

Also see:
  * http://msdn.microsoft.com/en-us/library/windows/desktop/ms724872(v=vs.85).aspx
  * http://code.google.com/p/libregf/downloads/detail?name=Windows%20NT%20Registry%20File%20%28REGF%29%20format.pd

Original comment by joachim.metz@gmail.com on 20 Nov 2012 at 10:00

[GoogleCodeExporter]

f missing in the last URL:
http://code.google.com/p/libregf/downloads/detail?name=Windows%20NT%20Registry%2
0File%20%28REGF%29%20format.pdf

Original comment by joachim.metz@gmail.com on 20 Nov 2012 at 10:00

[GoogleCodeExporter]

Thanks for the info, Joachim!  I noticed that note in Tim's documentation about 
the Major/Minor version and started trying to figure out how to check that in 
memory.. but I didn't realize that different registries would have different 
versions like that.  I'm glad to have more validation, however :-)

Thanks again for the information!

Original comment by jamie.l...@gmail.com on 20 Nov 2012 at 10:20

[GoogleCodeExporter]

Just adding a note: the version appears to be found in _HHIVE (the member is 
"Version"). modifying hivelist to print this out I get:

$ ./vol.py -f XPSP3.bin hivelist
Volatile Systems Volatility Framework 2.3_alpha
Virtual    Physical   Version Name
---------- ---------- ------- ----
0xe2975b60 0x24f64b60       3 \Device\HarddiskVolume1\Documents and 
Settings\user\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat
0xe2e08b60 0x3ac96b60       3 \Device\HarddiskVolume1\Documents and 
Settings\user\NTUSER.DAT
0xe2691b60 0x21225b60       3 \Device\HarddiskVolume1\Documents and 
Settings\LocalService\Local Settings\Application 
Data\Microsoft\Windows\UsrClass.dat
0xe2696008 0x21299008       3 \Device\HarddiskVolume1\Documents and 
Settings\LocalService\NTUSER.DAT
0xe2688658 0x212b3658       3 \Device\HarddiskVolume1\Documents and 
Settings\NetworkService\Local Settings\Application 
Data\Microsoft\Windows\UsrClass.dat
0xe2620380 0x206e7380       3 \Device\HarddiskVolume1\Documents and 
Settings\NetworkService\NTUSER.DAT
0xe1e13758 0x19773758       5 
\Device\HarddiskVolume1\WINDOWS\system32\config\software
0xe1e25008 0x1978f008       5 
\Device\HarddiskVolume1\WINDOWS\system32\config\default
0xe1e12758 0x19792758       3 
\Device\HarddiskVolume1\WINDOWS\system32\config\SECURITY
0xe1e14b60 0x1979ab60       3 
\Device\HarddiskVolume1\WINDOWS\system32\config\SAM
0xe16e9758 0x0ba8c758       3 [no name]
0xe1036b60 0x0b3d4b60       5 
\Device\HarddiskVolume1\WINDOWS\system32\config\system
0xe1024b60 0x0b387b60       3 [no name]
0x8067e1a8 0x0067e1a8       0 [no name]

./vol.py -f win7.dmp --profile=Win7SP0x86 hivelist
Volatile Systems Volatility Framework 2.3_alpha
Virtual    Physical   Version Name
---------- ---------- ------- ----
0x82b7a140 0x02b7a140 67...15 [no name]
0x820235c8 0x203675c8       3 \SystemRoot\System32\Config\SAM
0x87a0c428 0x27fe7428       3 [no name]
0x87a1a250 0x27eb3250       5 \REGISTRY\MACHINE\SYSTEM
0x87a429d0 0x27f9d9d0       3 \REGISTRY\MACHINE\HARDWARE
0x87ac34f8 0x135804f8       5 \SystemRoot\System32\Config\DEFAULT
0x88603008 0x20d36008       3 
\??\C:\Windows\ServiceProfiles\NetworkService\NTUSER.DAT
0x88691008 0x1ca1c008       3 
\??\C:\Windows\ServiceProfiles\LocalService\NTUSER.DAT
0x8b421908 0x26ab7908       3 \Device\HarddiskVolume1\Boot\BCD
0x8b42d558 0x2571c558       5 \SystemRoot\System32\Config\SOFTWARE
0x8cffb008 0x12424008       3 \??\C:\Users\user\ntuser.dat
0x8f3a19d0 0x219ff9d0       3 \SystemRoot\System32\Config\SECURITY
0x90a6e008 0x0e948008       3 
\??\C:\Users\user\AppData\Local\Microsoft\Windows\UsrClass.dat
0x9141e9d0 0x1dc569d0       3 \??\C:\Windows\System32\config\COMPONENTS

Original comment by jamie.l...@gmail.com on 21 Nov 2012 at 12:54

[GoogleCodeExporter]

Tha patch works good for me.
Thanks ;)

Original comment by xyzt...@gmail.com on 21 Nov 2012 at 4:27

[GoogleCodeExporter]

Can this patch be applied or do we need additional testing/documentation? 

Original comment by michael.hale@gmail.com on 3 Jan 2013 at 11:17

[GoogleCodeExporter]

I was actually going to write up something a little better using the version 
number.  Sorry it's taking a bit (bogged down) but I haven't forgotten.  Will 
have it soon.

Original comment by jamie.l...@gmail.com on 3 Jan 2013 at 11:25

[GoogleCodeExporter]

Original comment by michael.hale@gmail.com on 1 Feb 2013 at 4:15

• Added labels: Milestone-2.3.x

[GoogleCodeExporter]

I've decided to put this off until 3.0 since we are considering a rewrite of 
the registry stuff anyway.

Original comment by jamie.l...@gmail.com on 18 Feb 2013 at 10:02

• Added labels: Milestone-3.0
• Removed labels: Milestone-2.3.x

[GoogleCodeExporter]

placing all registry issues together

Original comment by jamie.l...@gmail.com on 7 Mar 2014 at 4:34

• Changed state: Duplicate

[GoogleCodeExporter]

fixed in 2.4

Original comment by jamie.l...@gmail.com on 3 Jul 2014 at 7:23

• Changed state: Fixed