search

stephanelpaul / volatility

Automatically exported from code.google.com/p/volatility
GNU General Public License v2.0
0 stars 0 forks source link

Port carl pulley's stack and heap parsing plugins to 2.0 #149

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Since scudette got me started...we should port the heap and stack plugins to 2.0

https://github.com/carlpulley/volatility/tree/master/memory_plugins/honeynet

don't worry about the file objects plugin for now.

Original issue reported on code.google.com by michael.hale@gmail.com on 7 Oct 2011 at 2:01

GoogleCodeExporter commented 9 years ago

Original comment by michael.hale@gmail.com on 7 Oct 2011 at 2:01

GoogleCodeExporter commented 9 years ago 
Due to some porting activity, the above link above should now be:

https://github.com/carlpulley/volatility/tree/v1.3/memory_plugins/honeynet

Original comment by carl.pulley on 9 Oct 2011 at 1:08

GoogleCodeExporter commented 9 years ago 
Thanks for the update Carl. I think someone got started on porting the heaps 
plugin already. I'll ask him to post the current draft here for feedback. 

Nice work on the plugins and honeynet challenge!

Original comment by michael.hale@gmail.com on 9 Oct 2011 at 2:46

GoogleCodeExporter commented 9 years ago 
That'd be great.

I'm planning on doing a bit more porting work over the next week (mainly with 
the exportfile plugin - I want to include some code for extracting memory 
mapped files from _FILE_OBJECT's). If time is left, I'll use it to start work 
on porting the stack plugin (I have a number of enhancements for it that I'll 
consider once the basic code is ported).

Thanks for the interest and help.

Original comment by carl.pulley on 9 Oct 2011 at 3:45

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Bug fixed version, hopefully. Also in plugin format. Sorry about the hassle 
earlier.

Original comment by marko.th...@gmail.com on 14 Oct 2011 at 9:27

Attachments:

GoogleCodeExporter commented 9 years ago 
Inspiring to see how nicely my ugly heaps code has been cleaned up. Nice work 
Marko.

The basic (but ugly!) stack code has also now been ported across to 2.0 (see 
https://github.com/carlpulley/volatility/blob/master/exportstack.py) - sorry, 
but this code is not as clean as Marko's. It is also worth noting that this 
plugin has a number of overlaps with MHL's threads plugin (primary differences: 
threads has a nice tagging and search mechanism; my stacks code will attempt to 
unwind kernel and user mode stacks).

The file extraction plugin is almost fully functional now. I've just got to 
finish the code for reconstructing files from extracted memory pages (that will 
get completed this weekend). All memory pages belonging to shared cache and 
memory mapped files are now extracted and saved (I still need to look at 
private cache maps). There's also a --pool option to just dump any _FILE_OBJECT 
that you can find!

Original comment by carl.pulley on 14 Oct 2011 at 2:02

GoogleCodeExporter commented 9 years ago 
updated version of heap plugin

Original comment by marko.th...@gmail.com on 13 Nov 2011 at 9:34

Attachments:

GoogleCodeExporter commented 9 years ago 
Fixed version of the heaps plugin.

Original comment by marko.th...@gmail.com on 15 Nov 2011 at 6:45

Attachments:

GoogleCodeExporter commented 9 years ago 
Has there been any work on cleaning up the stack plugin? I have been 
experimenting with it and it's old stack-frame lookup function could use some 
love. 

First, stack frames seems to be 4-byte aligned on 32-bit machines and 16-byte 
aligned on 64-bit machines, which would speed up the old stack frame search. 
(See  http://msdn.microsoft.com/en-us/library/aa290049(v=vs.71).aspx).

Furthermore, the current old stack-frame search skips searching for an EBP 
value once one old stack frame was found pointing to that EBP. There may other 
old frames pointing to the same EBP value if the frame called multiple 
functions that have all returned, and one of those functions had a parameter 
list larger than the entire other function (not sure how this will look):

                     old_n          old_n+1
                                   --------
                                   |  ebp    <- we could still find this
old esp unknown->  ---------       |  eip
                   | local         | input   <- from here on down it's overwritten by old_n
                   |  ebp          | input
                   |  eip          | input
                   | return        | return
             esp-> ---------       --------
                   | current       | current

By searching for both EBP's we can find more old stack frames. For example:

Thread @ 0x80f57228. PPID: 480. TID: 584. CrossThreadFlags: 0
    Initial Stack: [KERNEL]
    Stack: 0xfc2b2000-0xfc2b5000. Size: 0x3000
    Teb @ 0x7ffdb000. User Stack: 0x65c000-0x660000. Size: 0x4000
    Trapframe @ 0xfc2b4d64
        EDX: 0x9ac80. ECX: 0xffffffff. EAX: 0xc0. EDI: 0x1. ESI: 0x0. EBX: 0x0
        EBP: 0x65ffb4. EIP: 0x7c90eb94. ERR: 0x0. ESP: 0x65fcec
        DR0: 0x0. DR1: 0x0. DR2: 0x0. DR3: 0x0. DR6: 0x0. DR7: 0x0.
    [Current+1]  EIP: 0x7c90eb94. EBP: 0x65fad4 -> 0x65ffb4.
    [Current+2]  EIP: 0x7c90d625. EBP: 0x65fa0c -> 0x65ffb4.
    [Current+3]  EIP: 0x7c90e57c. EBP: 0x65f9ec -> 0x65fa0c.
    [Current]    EIP: 0x7c90eb94. EBP: 0x65ffb4
    [Current-1]  EIP: 0x7c80b50b. EBP: 0x65ffec.

Note how [Current+1] and [Current+2] were both called by the [Current] frame.

Original comment by tamas.k....@gmail.com on 20 Mar 2013 at 5:13

GoogleCodeExporter commented 9 years ago 
Hi Tamas,
  I'll look into incorporating these ideas into the exportstack.py code over the next week (I'll report back as soon as I have something viable).

Thank you very much for your feedback,

  Carl.

Original comment by carl.pulley on 20 Mar 2013 at 10:45

GoogleCodeExporter commented 9 years ago 
While you at it, I saw a #FIXME for the kernel ESP values and noticed you were 
not reading the kernel EIP and EBP values either. I've been just poking around 
and came across this: 
http://blog.airesoft.co.uk/2009/02/grabbing-kernel-thread-contexts-the-process-e
xplorer-way. It's not entirely clear in this article, but based on it I seem to 
have been able to also gather the kernel stack's EBP and EIP values, which do 
help in unwinding the kernel stack:

esp=ethread.Tcb.KernelStack.v()
eip=self.flat_address_space.read_long(self.process_address_space.vtop(esp+8))
ebp=self.flat_address_space.read_long(self.process_address_space.vtop(esp+12))

Thread @ 0x80f97020. PPID: 1540. TID: 1912. CrossThreadFlags: 0
    Initial Stack: [KERNEL]
    Stack: 0xfba49000-0xfba4c000. Size: 0x3000
    ESP: 0xfba4bc4c  EIP: 0x8054498b. EBP: 0xfba4bc98.
    [Current]    EIP: 0x8054498b. EBP: 0xfba4bc98 <- 0xfba4bd48.
    [Current-1]  EIP: 0x8054060c. EBP: 0xfba4bd48 <- 0xfba4bd64.
    [Current-2]  EIP: 0x7c90eb94. EBP: 0xfba4bd64 <- 0xd6ff80.

Original comment by tamas.k....@gmail.com on 21 Mar 2013 at 12:21

GoogleCodeExporter commented 9 years ago 
Also, in the stack_frame_iterator_up() function, you have:

ebp = self.flat_address_space.read_long(self.process_address_space.vtop(ebp))

This is fine for most cases, but the user stack can't be trusted to contain 
valid ebp's. The only checking you have right now is to check if the ebp is 
within the stack boundries (or if it's zero). What if for example the initial 
stack's EBP is overwritten to point back to the current frame? That would make 
the loop run forever.. so we may want to include some form of safety check to 
break when we hit already visited stack frames.

Original comment by tamas.k....@gmail.com on 21 Mar 2013 at 2:09

GoogleCodeExporter commented 9 years ago 
Just released a rewrite of the exportstack plugin that incorporates Tamas' 
suggestions. Plugin now:
  -extends malware.threads
  -unwinds the exception handling stack
  -has a good result correlation with WinDBG based stack unwinds.

Original comment by carl.pulley on 30 Mar 2013 at 7:54

GoogleCodeExporter commented 9 years ago 
Nice work Carl! The plugin works great.

Also, I was exploring the values found around the ESP a bit more as I can't 
find any more information on it. This is what I came up with so far:

esp = thread.Tcb.KernelStack.v()
*esp = points to an address on the kernel stack
*esp+4: 0x246, constant? I see it on both XP and Win7.
*esp+8: kernel EIP (according to link I already sent)
*esp+12: kernel EBP
*esp+16: thread base pointer
*esp+20: 0xffdff120 on XP
*esp+32: on XP pointer to WaitListBlock (thread base pointer+70).
*esp+36: thread base pointer

Now, if we read values lower from the esp, on XP:

*esp-4: thread base pointer or 0xffdff538
*esp-8: 0xffdff120
*esp-12: thread base pointer
*esp-16: 0x1f

On Win7:
*esp-4: thread base pointer
When *esp-8 is not 1, than *esp-16 is also the thread base pointer.

I figured I post it here in case it's useful for anything.

Original comment by tamas.k....@gmail.com on 31 Mar 2013 at 12:00

GoogleCodeExporter commented 9 years ago 
I think it might be worthwhile exploring the integration of the stack plugin 
with pdbparse's symbol lookup tool 
(https://code.google.com/p/pdbparse/source/browse/trunk/examples/pdb_lookup.py).
 It would enable us to translate the EIP values into symbols on the fly. The 
most practical approach would be to extract the GUID+pdb filename from the 
module's PE header the EIP maps to and download the pdb automatically to use 
with pdb_lookup.

For example I was able to get this:

Thread @ 0x853da908. PPID: 4. TID: 1828. CrossThreadFlags: 43010
    Start address: 0x8b51cd18
    Stack: 0x92009000-0x9200bed0. Size: 0x2ed0.
    Kernelstack @ 0x9200bae0
        ECX: 0x9200bc30. EBP: 0x9200bb30. EIP: 0x82686766. ESP: 0x9200bae0
        ntoskrnl.exe!@KiSwapContext@8
    [Current]    EIP: 0x8268c4f7.    EBP: 0x9200bb30 -> 0x9200bb58.  Size: 0x50
        ntoskrnl.exe!_KiCommitThreadWait@16
    [Current-1]  EIP: 0x8268d1ed.    EBP: 0x9200bb58 -> 0x9200bbbc.  Size: 0x28
        ntoskrnl.exe!_KeRemoveQueueEx@24
    [Current-2]  EIP: 0x826a732e.    EBP: 0x9200bbbc -> 0x9200bbdc.  Size: 0x64
        ntoskrnl.exe!_KeRemoveQueue@12
    [Current-3]  EIP: 0x8b50708e.    EBP: 0x9200bbdc -> 0x9200bc40.  Size: 0x20
        rdbss.sys!_RxpWorkerThreadDispatcher@8
    [Current-4]  EIP: 0x8b51cd32.    EBP: 0x9200bc40 -> 0x9200bc50.  Size: 0x64
        rdbss.sys!_RxWorkerThreadDispatcher@4
    [Current-5]  EIP: 0x82818f5e.    EBP: 0x9200bc50 -> 0x9200bc90.  Size: 0x10
        ntoskrnl.exe!_PspSystemThreadStartup@8
    [Current-6]  EIP: 0x826c0219.    EBP: 0x9200bc90 -> 0x0.     Size: 0x40
        ntoskrnl.exe!_KiThreadStartup@4

For non-kernel processes it seems to work just as well but some modules don't 
seem to have a IMAGE_DIRECTORY_ENTRY_DEBUG loaded in memory (win32k.sys and 
ntdll.dll), so I can't get the GUID and consequently the pdb:

Target pid: 200
Walking STACKS of smss.exe
Thread @ 0x84b46930. PPID: 200. TID: 204. CrossThreadFlags: 43074
    Start address: 0x76ea7098
    Stack: 0x88bd4000-0x88bd6ed0. Size: 0x2ed0.
    Kernelstack @ 0x88bd6648
        ECX: 0x88bd6ab8. EBP: 0x88bd6698. EIP: 0x82686766. ESP: 0x88bd6648
        ntoskrnl.exe!@KiSwapContext@8
    [Current]    EIP: 0x8268c4f7.    EBP: 0x88bd6698 -> 0x88bd66c0.  Size: 0x50
        ntoskrnl.exe!_KiCommitThreadWait@16
    [Current-1]  EIP: 0x826884a4.    EBP: 0x88bd66c0 -> 0x88bd683c.  Size: 0x28
        ntoskrnl.exe!_KeWaitForMultipleObjects@32
    [Current-2]  EIP: 0x82839796.    EBP: 0x88bd683c -> 0x88bd6ac8.  Size: 0x17c
        ntoskrnl.exe!_ObpWaitForMultipleObjects@20
    [Current-3]  EIP: 0x82839503.    EBP: 0x88bd6ac8 -> 0x88bd6c18.  Size: 0x28c
        ntoskrnl.exe!_NtWaitForMultipleObjects@20
    [Current-4]  EIP: 0x8264d1ea.    EBP: 0x88bd6c18 -> 0x88bd6c34.  Size: 0x150
        ntoskrnl.exe!_KiFastCallEntry
    [Current-5]  EIP: 0x76ea70b4.    EBP: 0x88bd6c34 -> 0x14fef8.    Size: 0x1c
        ntdll.dll!(null)
    Teb @ 0x7ffdf000. User Stack: 0x14d000-0x150000. Size: 0x3000
    Trapframe @ 0x88bd6c34
        EDX: 0x2529c8. ECX: 0x0. EAX: 0x0. EDI: 0x25e760. ESI: 0x0. EBX: 0x7fff00000000
        EBP: 0x14fef8. EIP: 0x76ea70b4. ERR: 0x0. ESP: 0x14fe44
        DR0: 0x0. DR1: 0x0. DR2: 0x0. DR3: 0x0. DR6: 0x0. DR7: 0x7fff00000000.
        ntdll.dll!(null)
    [Current]    EIP: 0x4821e946.    EBP: 0x14fef8 -> 0x14ff3c.  Size: 0xb4
        smss.exe!_NtProcessStartupW_AfterSecurityCookieInitialized@4
    [Current-1]  EIP: 0x76e65e7a.    EBP: 0x14ff3c -> 0x14ff7c.  Size: 0x44
        ntdll.dll!(null)
    [Current-2]  EIP: 0x76ec37c8.    EBP: 0x14ff7c -> 0x14ff94.  Size: 0x40
        ntdll.dll!(null)
    [Current-3]  EIP: 0x0.       EBP: 0x14ff94 -> 0x0.   Size: 0x18

Original comment by tamas.k....@gmail.com on 12 Apr 2013 at 10:39

GoogleCodeExporter commented 9 years ago 
In cases where IMAGE_DIRECTORY_ENTRY_DEBUG is paged out, you can also try
retrieving a copy of the PE based on its GUID, which is usually still paged
in, and then using that to get the PDB. I have some example code that does
this here:

http://18.26.6.153/~brendan/resolvetaps.py

-Brendan

Original comment by moo...@gmail.com on 12 Apr 2013 at 11:48

GoogleCodeExporter commented 9 years ago 
Hi Brendan, thanks, I'll check again! My understanding was (and correct me if 
I'm wrong) that if IMAGE_DIRECTORY_ENTRY_DEBUG is paged out I won't be able to 
get the GUID. When it's paged out the debug type is reported as Unknown type: 
IMAGE_DEBUG_TYPE_UNKNOWN.

Original comment by tamas.k....@gmail.com on 13 Apr 2013 at 1:40

GoogleCodeExporter commented 9 years ago 
The PDB's GUID may be paged out, but the PE's GUID, which is just the
PE's FileHeader.TimeDateStamp
combined with its OptionalHeader.SizeOfImage. Since both of those are
generally in the first page of the executable they're usually in memory, so
you can use them to get the full PE from Microsoft's symbol server. The
process goes:

1. Try to get the PDB using IMAGE_DIRECTORY_ENTRY_DEBUG. If that fails,
2. Get the PE's TimeDateStamp+SizeOfImage and use it to download the full
PE from MS.
3. Use the downloaded PE's IMAGE_DIRECTORY_ENTRY_DEBUG to download the PDB.

Hope that helps!

Original comment by moo...@gmail.com on 13 Apr 2013 at 2:32

GoogleCodeExporter commented 9 years ago 
Hi Tamas and Brenden,
  I have some code that does exactly this - I will be able to release it within a week or so. I've some additional work to do so that functional ordinals can be better resolved (some process address spaces don't have the full set of module function names loaded, but often other process address spaces do - I need to finish unifying this collective data).

I also have some code that allows alternative stack unwind strategies to be 
applied (e.g. so that CALL/RETs, ROP gadgets, etc. can be detected). This is 
done in a manner similar to malware.thread tags and so should be extensible.

I'll report back once I'm done.

All the best,

  Carl.

Original comment by carl.pulley on 13 Apr 2013 at 8:53

GoogleCodeExporter commented 9 years ago 
Brendan: cool thanks, that makes sense. I'll give it a shot!

Carl: great, I'm very interested in the alternative unwind strategies =) When 
you say "some process address spaces don't have the full set of module function 
names loaded", do you mean the module export tables? The problem with the 
export tables are, well, the unexported functions for which we need the debug 
info.

Original comment by tamas.k....@gmail.com on 13 Apr 2013 at 1:12

GoogleCodeExporter commented 9 years ago 
Hi Tamas,
  currently I perform addr to symbol resolution as follows:
    -determine module that the address lies in
    -if PDB symbol resolution is possible (and the user hasn't disallowed it), download (if necessary) and load PDB symbol info. and resolve addr to nearest debugging symbol name (c.f. the process Brendan has described)
    -if PDB symbol resolution is not possible, use the module exports to resolve the addr.

It currently looks like the exports function name information can vary between 
process address spaces (here I'm thinking about processes that both load a 
common module). So, in the failover case above, I need to unify this 
information and so improve the accuracy of the exports-based resolution process.

Hopefully that clarifies the issue?

  Carl.

Original comment by carl.pulley on 13 Apr 2013 at 1:53

GoogleCodeExporter commented 9 years ago 
Hi Carl, thanks, yes, that sounds like a good approach. Brendan's trick to get 
the full PE if the PDB guid is not available was a really nice hint =)

    Trapframe @ 0x94acfc34
        EDX: 0x0. ECX: 0x0. EAX: 0x76e903e7. EDI: 0x251c18. ESI: 0x2. EBX: 0x7fff00251c18
        EBP: 0x1bfa20. EIP: 0x76ea70b4. ERR: 0x0. ESP: 0x1bf8c0
        DR0: 0x0. DR1: 0x0. DR2: 0x0. DR3: 0x0. DR6: 0x0. DR7: 0x7fff00000000.
        ntdll.dll!_KiFastSystemCallRet@0
    [Current]    EIP: 0x76e65e7a.    EBP: 0x1bfa20 -> 0x1bfa60.  Size: 0x160
        ntdll.dll!_wcsspn
    [Current-1]  EIP: 0x76ec37c8.    EBP: 0x1bfa60 -> 0x1bfa78.  Size: 0x40
        ntdll.dll!__RtlUserThreadStart@8
    [Current-2]  EIP: 0x0.       EBP: 0x1bfa78 -> 0x0.   Size: 0x18

Original comment by tamas.k....@gmail.com on 13 Apr 2013 at 11:11

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Hi,
  just released some updates to the exportstack code. Stack frame unwinding is more modular and alternative unwind strategies can now be added. I've got implementations for the following types of unwinds:

  -EBP/ESP frame unwinds
  -CALL/RET unwinds
  -exception frame unwinds
  -prototype GS based extension to EBP/ESP unwinds (still has some kinks to work out)
  -prototype gadget detection (not yet tested)

I haven't (yet) got any of the newer 64 bit based unwinds implemented (I still 
need to look over some GDB sources etc. here).

Experimental (yara based) stack frame search functionality is injected into the 
_EPROCESS object class.

I'm still not happy with performance, so we'll see what can be done here with 
time.

All the best,

  Carl.

Original comment by carl.pulley on 18 May 2013 at 9:38

GoogleCodeExporter commented 9 years ago 
Hi guys, just doing some issue tracker clean up here. For now, I think it seems 
reasonable that the authors of the plugins (Carl and Tamas) maintain these 
plugins in their own code repositories. If it becomes a hassle for people to 
download plugins from multiple sources, then we can reconsider bringing them 
into the core code base. If you prefer to integrate them sooner than later, 
feel free to reopen the issue and attach the most recent version of your plugin 
here and we'll help to get it tested/applied. Thanks guys!

Original comment by michael.hale@gmail.com on 7 Mar 2014 at 5:12