Add Windows commandline history records plugin

[GoogleCodeExporter]

In Eoghan Casey’s 2010 article: "Extracting Windows command line details from 
physical memory", there is a plugin cmd_history.py.

We should add this to core plugins.

Apparently someone has the code for a previous version of volatility and it 
needs to be updated. Could they please attach to this issue and we can track 
progress of it here.

Original issue reported on code.google.com by scude...@gmail.com on 7 Oct 2011 at 7:33

[GoogleCodeExporter]

I've mostly ported this to 2.0 already in two versions (list walking and 
scanning). I was waiting for two things to happen before sharing:

1) applying the vad.get_data() patch (this already happened)
2) PDB support (needed for list walking version)

It may need some cleaning up, but give me a few and I'll attach it here. 

Original comment by michael.hale@gmail.com on 7 Oct 2011 at 1:55

[GoogleCodeExporter]

Original comment by mike.auty@gmail.com on 7 Oct 2011 at 7:59

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Here's a draft for you guys to poke through. The original plugin by Casey was 
for XP and volatility 1.3. It was a discontig scanner that looked for \x32\x00 
(a member of _COMMAND_HISTORY) and then appied some sanity checks around those 
two bytes. Next it searched for the same patters in csrss.exe and tried to do 
vtop translation to see if the patterns found by discontig scanner were in the 
memory space of csrss.exe. If not, it ignored the findings. So you'll see I've 
done some drastically different things in my plugins. 

The plugins work on all OS that Volatility supports (XP, 2003, 2008, Vista, 7) 
and Volatility 2.0.

They skip the discontig physical scanner and go straight to the vad of 
csrss.exe (XP) or conhost.exe (Windows 7 and later). This doesn't break 
anything since the findings from physical space are ignored if they're not in 
csrss.exe anyway.

They allow \x32\x00 to be tweaked on command line since that value can be 
changed programatically via Windows API or by the user by changing the cmd.exe 
properties. 

The Consoles command is completely different than the method described in 
Casey's paper. Instead of scanning for _COMMAND_HISTORY it scans for 
_CONSOLE_INFORMATION (which has a linked list to _COMMAND_HISTORY). This 
command only works for active consoles/command shells. However, it prints some 
pretty awesome information, including the entire screen buffer (both input and 
output of the cmd.exe console), command aliases, console titles, etc.  

Finally, the CmdList plugin is just a proof-of-concept and relies on PDB 
support. It prints all the same info as Consoles, but just finds the structures 
differently (you'll see the FIXME statements in there). We'll probably want to 
remove CmdList until PDB support exists. 

I'm hoping you all can test these out and provide some feedback. Also, if you 
see anything that can be cleaned up, please feel free to suggest or just attach 
a patched version - especially those of you who know python and volatility's 
object stuff way better than me. 

I've also attached a comparison of Casey's plugin (cmd_history) with the output 
of CmdScan and Consoles. 

Original comment by michael.hale@gmail.com on 11 Oct 2011 at 8:27

Attachments:

• cmdhistory.py
• command_history.txt

[GoogleCodeExporter]

Anyone have a chance to try this out? I figured you'd be all over this scudette 
since you asked for it ;-)

Original comment by michael.hale@gmail.com on 27 Oct 2011 at 11:54

[GoogleCodeExporter]

Well, I've only got a couple of images.  One XP one, where both plugins turn up 
nothing, and a windows 7 one where the cmdscan works great (and shows me 
running cygwin's sshd wrapper) but cmdlist gives the following:

[conhost.exe @ 0x84cc9570 pid 1668]
  [history @ 0xc581c0 0x1758e1c0]
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/home/mike/workspace/volatility/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/home/mike/workspace/volatility/volatility/plugins/cmdhistory.py", line 418, in render_text
    outfd.write("    CommandCount: {0}\n".format(hist.CommandCount))
  File "/home/mike/workspace/volatility/volatility/obj.py", line 777, in __getattr__
    return self.m(attr)
  File "/home/mike/workspace/volatility/volatility/obj.py", line 762, in m
    raise AttributeError("Struct {0} has no member {1}".format(self.obj_name, attr))
AttributeError: Struct _CONSOLE_INFORMATION has no member CommandCount

They're both x86 if that helps?

Original comment by mike.auty@gmail.com on 23 Jan 2012 at 8:36

[GoogleCodeExporter]

Thanks for testing Mike! There are 3 plugins here: cmdscan, consoles, and 
cmdlist. As stated above, don't use cmdlist ("Finally, the CmdList plugin is 
just a proof-of-concept and relies on PDB support. It prints all the same info 
as Consoles, but just finds the structures differently (you'll see the FIXME 
statements in there). We'll probably want to remove CmdList until PDB support 
exists.").

One thing that really bothers me about this plugin is this:

class UnicodeString(basic.String):

    def __str__(self):
        data = self.v()
        result = data.split("\x00\x00")[0]
        if not result:
            return ""
        #return result
        return repr(result.replace("\x00",""))

Its the reason I wrote you guys about how best to handle the LPWSTR data types. 
So I think we should:

* Delete the cmdlist plugin until PDB support is available 
* Fix the LPWSTR issue 
* I'll give it another round of cleaning up and testing soon

Original comment by michael.hale@gmail.com on 24 Jan 2012 at 8:10

[GoogleCodeExporter]

My bad. I asked for this issue to be added, and never noticed when it was 
updated (I really need to start using Volatility more instead of Memoryze). 
I've just downloaded the test code and tried it out against one of the hogfly 
test memory dumps at 
https://cid-5694a755c9c6a175.skydrive.live.com/browse.aspx/Public. Both new 
plugins appear to crash

root@SIFT-Workstation:/root/Desktop/volatility-2.0# python vol.py imageinfo -f 
/home/sansforensics/exemplar12_2.vmem
Volatile Systems Volatility Framework 2.0
Determining profile based on KDBG search...

          Suggested Profile(s) : WinXPSP3x86, WinXPSP2x86 (Instantiated with WinXPSP2x86)
                     AS Layer1 : JKIA32PagedMemoryPae (Kernel AS)
                     AS Layer2 : FileAddressSpace (/home/sansforensics/exemplar12_2.vmem)
                      PAE type : PAE
                           DTB : 0x7d0000
                          KDBG : 0x80544ce0L
                          KPCR : 0xffdff000L
             KUSER_SHARED_DATA : 0xffdf0000L
           Image date and time : 2009-01-08 01:57:04 
     Image local date and time : 2009-01-08 01:57:04 
          Number of Processors : 1
                    Image Type : Service Pack 2
root@SIFT-Workstation:/root/Desktop/volatility-2.0# 

root@SIFT-Workstation:/root/Desktop/volatility-2.0# python vol.py cmdscan -f 
/home/sansforensics/exemplar12_2.vmem
Volatile Systems Volatility Framework 2.0
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/root/Desktop/volatility-2.0/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 412, in render_text
    for task, hist in data:
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 356, in calculate
    data = vad.get_data()
  File "/root/Desktop/volatility-2.0/volatility/obj.py", line 301, in __getattr__
    raise AttributeError("Unable to resolve attribute {0} on {1}".format(attr, self.obj_name))
AttributeError: Unable to resolve attribute get_data on VadRoot
root@SIFT-Workstation:/root/Desktop/volatility-2.0# python vol.py consoles -f 
/home/sansforensics/exemplar12_2.vmem
Volatile Systems Volatility Framework 2.0
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/root/Desktop/volatility-2.0/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 518, in render_text
    for task, console in data:
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 481, in calculate
    data = vad.get_data()
  File "/root/Desktop/volatility-2.0/volatility/obj.py", line 301, in __getattr__
    raise AttributeError("Unable to resolve attribute {0} on {1}".format(attr, self.obj_name))
AttributeError: Unable to resolve attribute get_data on VadRoot

root@SIFT-Workstation:/root/Desktop/volatility-2.0# python vol.py cmdscan -f 
/home/sansforensics/exemplar12_2.vmem
Volatile Systems Volatility Framework 2.0
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/root/Desktop/volatility-2.0/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 412, in render_text
    for task, hist in data:
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 356, in calculate
    data = vad.get_data()
  File "/root/Desktop/volatility-2.0/volatility/obj.py", line 301, in __getattr__
    raise AttributeError("Unable to resolve attribute {0} on {1}".format(attr, self.obj_name))
AttributeError: Unable to resolve attribute get_data on VadRoot
root@SIFT-Workstation:/root/Desktop/volatility-2.0# python vol.py consoles -f 
/home/sansforensics/exemplar12_2.vmem
Volatile Systems Volatility Framework 2.0
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/root/Desktop/volatility-2.0/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 518, in render_text
    for task, console in data:
  File "/root/Desktop/volatility-2.0/volatility/plugins/cmdhistory.py", line 481, in calculate
    data = vad.get_data()
  File "/root/Desktop/volatility-2.0/volatility/obj.py", line 301, in __getattr__
    raise AttributeError("Unable to resolve attribute {0} on {1}".format(attr, self.obj_name))
AttributeError: Unable to resolve attribute get_data on VadRoot

Can you guys replicate this?
Thanks
John

Original comment by johnmcc...@comcast.net on 4 Mar 2012 at 8:53

[GoogleCodeExporter]

Strangely, it appears to work on the version of 2.1 alpha that's shipped with 
the SIFT Kit (Wow! Way Coool, BTW), but not with production 2.0 or with the 
current 2.1 svn.
John

Original comment by johnmcc...@comcast.net on 5 Mar 2012 at 4:41

[GoogleCodeExporter]

Hmmm, well, I just tried it and apart from a slight issue with the backwards 
compatibility of the new profile innards, it all seemed to work ok?  I 
certainly wasn't getting any problems with VadRoot having no get_data attribute.

Could you please try getting a clean subversion copy (at least up to r1508 to 
include the profile compatibility fix I just added), and then add just the 
cmdhistory.py from comment 3?  If that all works fine, then it may be you have 
another plugin overriding the normal VadRoot plugin.  If it doesn't work then 
something very weird's going on and we can investigate further...  5:)

Original comment by mike.auty@gmail.com on 5 Mar 2012 at 10:16

[GoogleCodeExporter]

OK,
I pulled down the most recent svn version, and did one quick test, and it 
appeared to work OK. I'm kind of swamped this week, but will try to do some 
more extensive testing. Question: What exactly are the structures (for someone 
who has only the most tenuous grasp of Windows internals and can't pry loose a 
couple of hours to research it right now) from which the two working plugins 
and the one to be enabled extract the information? So far I've had a single 
(rather spectacular, mostly because of the screen buffer that was ectracted) 
success with the consoles plugin, and most other places I've tried it I've 
found essentially nothing.

Also, on the image I had success with (this is the image from last year which 
prompted the initial search to find Casey's plugin, and led to my original 
complaint about it not being available) I found other commands that I believe 
were executed in cmd sessions doing string analysis. Maybe the plugin to be 
added later would have found these commands as well? I believe that the memory 
image I captured in that case was actually obtained while an attacker shell 
session was actively open. Which, if any, of the structures this information is 
being parsed out from would be expected to survive the termination of their 
host shell process for any significant period of time, assuming no other 
significant processing activity was going on?

Thanks
John

Original comment by johnmcca...@gmail.com on 9 Mar 2012 at 12:09

[GoogleCodeExporter]

Just a note for when these get committed, they need to use _EPROCESS.get_vads 
rather than the deprecated automatic address space switching that used to go on 
with .VadRoot (see r1548).

Original comment by mike.auty@gmail.com on 13 Mar 2012 at 1:35

[GoogleCodeExporter]

Wow sorry guys, I completely missed this discussion that John started back in 
March 4th. I just added myself to the watchers for this issue, so that won't 
happen again. 

Anyway, now that I see it, and more importantly - now that the patch for better 
unicode string handling has been committed, this plugin is next on my list to 
fix up. Please stay tuned!

Original comment by michael.hale@gmail.com on 23 Mar 2012 at 6:30

[GoogleCodeExporter]

Hey folks, 

Here's the cmdscan and consoles plugins ported to 2.1 and x64. I've tested it 
on at least one memory image from every x86 and x64 windows OS that volatility 
supports. It currently does not work on win7x64, but all others should be okay. 

> Question: What exactly are the structures (for someone who has only the most 
tenuous 
> grasp of Windows internals and can't pry loose a couple of hours to research 
it right now) 
> from which the two working plugins and the one to be enabled extract the 
information?

Probably the best reference is just MSDN's Console section: 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682055(v=vs.85).aspx 
and Stevens & Casey's original DFRWS paper: 
http://ww.dfrws.org/2010/proceedings/stevens.pdf

MSDN documents some of the APIs involved in creating and using consoles, but 
the internal structures are not documented. I do plan on writing something up 
on it, but for now the best documentation I have is the attached plugin (the 
structure names, offsets, members, and relationships are all commented pretty 
well though). 

> Which, if any, of the structures this information is being parsed out from 
would be 
> expected to survive the termination of their host shell process for any 
significant period
> of time, assuming no other significant processing activity was going on?

I'm afraid I don't have a great answer for that, but Stevens & Casey's paper 
might give you some idea. I've just RE'd the structures and written up the best 
way I know to locate them. One thing for sure is that the data is going to 
survive longer on xp/2003/vista/2008 than on win7. That's because 
xp/2003/vista/2008 history is all stored in csrss.exe which is always running 
while the system is running, but with win7 each cmd.exe can have a separate 
conhost.exe for shell/console hosting. When cmd.exe dies, so does conhost.exe. 
In other words, console processses in general on win7 have much shorter 
lifespans. 

If you have any other questions, feel free to ask, and certainly feel free to 
offer results of your testing ;-)

Original comment by michael.hale@gmail.com on 1 Apr 2012 at 6:17

Attachments:

• cmdhistory.py

[GoogleCodeExporter]

This issue was closed by revision r1592.

Original comment by michael.hale@gmail.com on 4 Apr 2012 at 7:22

• Changed state: Fixed

[GoogleCodeExporter]

Hi,
I just noticed this was closed out. I tested the new consoles command in rev 
1794 under both the SIFT kit VM and Cygwin on my Win7x64 workstation. It seems 
to have a problem:

$ python ./vol.py consoles -f 
//APT_NAS_1/share/INCD-2010-10-00021/P22012-06/P22012-06_PhysicalMemory2.img
Volatile Systems Volatility Framework 2.1_alpha
**************************************************
ConsoleProcess: csrss.exe Pid: 1720
Console: 0x4f23a8 CommandHistorySize: 50
HistoryBufferCount: 1 HistoryBufferMax: 4
OriginalTitle: %SystemRoot%\Microsoft.NET\Framework\v2.0.50727\aspnet_wp.exe
Title: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_wp.exe
AttachedProcess: aspnet_wp.exe Pid: 6052 Handle: 0x148c
----
CommandHistory: 0xf986f8 Application: aspnet_wp.exe Flags: Allocated
CommandCount: 0 LastAdded: -1 LastDisplayed: -1
FirstCommand: 0 CommandCountMax: 50
ProcessHandle: 0x148c
Traceback (most recent call last):
  File "./vol.py", line 185, in <module>
    main()
  File "./vol.py", line 176, in main
    command.execute()
  File "/home/Forensic/newvol/volatility/commands.py", line 109, in execute
    func(outfd, data)
  File "/home/Forensic/newvol/volatility/plugins/malware/cmdhistory.py", line 789, in render_text
    for screen in console.get_screens():
  File "/home/Forensic/newvol/volatility/plugins/malware/cmdhistory.py", line 390, in get_screens
    if self.ScreenBuffer not in screens:
  File "/home/Forensic/newvol/volatility/obj.py", line 534, in __eq__
    return (0xffffffffffff & self.v()) == (0xffffffffffff & other)
TypeError: unsupported operand type(s) for &: 'long' and 'Pointer'

Thoughts? Thanks.
John

Original comment by johnmcc...@comcast.net on 25 May 2012 at 11:46

[GoogleCodeExporter]

Hi John,
  >  File "/home/Forensic/newvol/volatility/obj.py", line 534, in __eq__
  >    return (0xffffffffffff & self.v()) == (0xffffffffffff & other)

Can you please try to replace that line with:
  >    return (0xffffffffffff & int(self)) == (0xffffffffffff & int(other))

Original comment by scude...@gmail.com on 25 May 2012 at 12:03

[GoogleCodeExporter]

With that modification, it no longer crashes.
Thanks much!
John

Original comment by johnmcc...@comcast.net on 25 May 2012 at 12:27

[GoogleCodeExporter]

John, you've hit upon a different bug.  If you'd like to follow it up, please 
see issue 265.

Original comment by mike.auty@gmail.com on 25 May 2012 at 10:21

[GoogleCodeExporter]

Hiya John, I've tried to recreate the problem you were encountering, but cannot 
figure out how it came about.  Going through the code, and running similar 
tests, volatility should handle both (long & Pointer) and (Pointer & long) 
without a problem.

Please could you verify that you're using trunk, and that you've either run 
"make clean" or deleted *.pyc from all volatility directories and 
subdirectories.  You also mentioned you'd tested this on SIFT kit VM and 
Cygwin, could you please attempt to verify whether you're seeing this behaviour 
under a normal windows or linux environment, and if so what version of python 
you're using?

We should use issue 265 to follow this up, hopefully we can figure out the 
strange discrepancy soon...

Original comment by mike.auty@gmail.com on 26 May 2012 at 9:35

[GoogleCodeExporter]

I know someone else who received this same error on an x86 windows host. I'll 
follow up after I get their python and volatility build/revision info. 

Original comment by michael.hale@gmail.com on 26 May 2012 at 4:06

[GoogleCodeExporter]

Alright the other case was using 2.7.1 on an x86 windows xp machine. Volatility 
r1791 from 5/22. 

Original comment by michael.hale@gmail.com on 29 May 2012 at 8:06

[GoogleCodeExporter]

Ok, thanks, could we please continue all conversation on issue 265, this one's 
closed and done.  

Original comment by mike.auty@gmail.com on 29 May 2012 at 8:09