Volatility 2.6 parsers are a problem after 4.9 kernels versions.

[cpuu]

root@kali:~/lmg/volatility# uname -a Linux kali 4.9.0-kali3-amd64 #1 SMP Debian 4.9.18-1kali1 (2017-04-04) x86_64 GNU/Linux

and i execute command right below root@kali:~/lmg/volatility# python vol.py --conf-file=../capture/kali-2017-05-03_01.40.13/volatilityrc linux_pslist

Volatility Foundation Volatility Framework 2.6
Traceback (most recent call last):
  File "vol.py", line 192, in <module>
    main()
  File "vol.py", line 183, in main
    command.execute()
  File "/root/lmg/volatility/volatility/plugins/linux/common.py", line 64, in execute
    commands.Command.execute(self, *args, **kwargs)
  File "/root/lmg/volatility/volatility/commands.py", line 116, in execute
    if not self.is_valid_profile(profs[self._config.PROFILE]()):
  File "/root/lmg/volatility/volatility/plugins/overlays/linux/linux.py", line 216, in __init__
    obj.Profile.__init__(self, *args, **kwargs)
  File "/root/lmg/volatility/volatility/obj.py", line 862, in __init__
    self.reset()
  File "/root/lmg/volatility/volatility/plugins/overlays/linux/linux.py", line 227, in reset
    self.load_vtypes()
  File "/root/lmg/volatility/volatility/plugins/overlays/linux/linux.py", line 264, in load_vtypes
    vtypesvar = dwarf.DWARFParser(dwarfdata).finalize()
  File "/root/lmg/volatility/volatility/dwarf.py", line 71, in __init__
    self.feed_line(line)
  File "/root/lmg/volatility/volatility/dwarf.py", line 162, in feed_line
    self.process_statement(**parsed) #pylint: disable-msg=W0142
  File "/root/lmg/volatility/volatility/dwarf.py", line 204, in process_statement
    self.vtypes[name] = [ int(data['DW_AT_byte_size'], self.base), {} ]
KeyError: 'DW_AT_byte_size'

is it a volatility problem? or dwarfdump bugs?

I think that volatility cannot parse the data structure which came from kernel later than 4.4 . my test on 4.4 is alright. but 4.9 doesn't work.

[bneuburg]

C.f. volatilityfoundation/volatility#335

That PR includes a fix for the dwarfdump parsing in newer kernel.

If you still have problems after applying that patch, those might be related to the changes in kernel address space layout randomization, c.f. volatilityfoundation/volatility#385

[cpuu]

@bneuburg you such a genius !! Thanks a lot! I changed the code like it

but linux_kaslr_shift plugin shows me 294lines of results. Which line is correct shift size? what should I choose?

root@kali:~/lmg/volatility# cat 1.txt
DTB              VirtualShift     Physical Shift  
       0x5007000   0xc3568091c000  -0xc3567d31c000
       0x5007000   0xc35680d80000  -0xc3567d780000
       0x5007000   0xc356816da000  -0xc3567e0da000
       0x5007000   0xc3568187b000  -0xc3567e27b000
       0x5007000   0x9fb9be600000  -0x9fb9bb000000
       0x5007000   0x9fb9be610000  -0x9fb9bb010000
       0x5007000   0x9fb9be620000  -0x9fb9bb020000
       0x5007000   0x9fb9be630000  -0x9fb9bb030000
       0x5007000   0x9fb9be640000  -0x9fb9bb040000
       0x5007000   0x9fb9be650000  -0x9fb9bb050000
       0x5007000   0x9fb9be660000  -0x9fb9bb060000
       0x5007000   0x9fb9be670000  -0x9fb9bb070000
       0x5007000   0x9fb9be680000  -0x9fb9bb080000

and so on 

[cpuu]

@bneuburg Testing on diffrent system

root@kali:~/lmg/volatility# python vol.py --conf-file=../capture/kali-2017-05-04_04.01.18/volatilityrc  linux_kaslr_shift
Volatility Foundation Volatility Framework 2.6
DTB              VirtualShift     Physical Shift  
      0x89607000       0x3c800000       0x4b400000

It works very well ! Thankyou

root@kali:~/lmg/volatility# python vol.py --conf-file=../capture/kali-2017-05-04_04.01.18/volatilityrc --physicalshift=1262485504 --dtb=0x89607000 --shift=0x3c800000 linux_pslist
Volatility Foundation Volatility Framework 2.6
Offset             Name                 Pid             PPid            Uid             Gid    DTB                Start Time
------------------ -------------------- --------------- --------------- --------------- ------ ------------------ ----------
0xffff9614daaf3040 systemd              1               0               0               0      0x000000011a6bb000 2017-05-04 07:54:12 UTC+0000
0xffff9614daafa080 kthreadd             2               0               0               0      ------------------ 2017-05-04 07:54:12 UTC+0000
0xffff9614daaff0c0 ksoftirqd/0          3               2               0               0      ------------------ 2017-05-04 07:54:12 UTC+0000
0xffff9614dab0a140 kworker/0:0H         5               2               0               0      ------------------ 2017-05-04 07:54:12 UTC+0000
0xffff9614dabb1040 rcu_sched            7               2               0               0      ------------------ 2017-05-04 07:54:12 UTC+0000

[bneuburg]

@cpuu: Can you attach 1.txt? When in doubt I'd try the row where the number of trailing 0's for virtual and physical shift aggregated is the highest. In the snippet you pasted that would be

0x5007000   0x9fb9be600000  -0x9fb9bb000000

What Linux distribution was this?

[cpuu]

1.txt

This is Kali Linux (for Pentest), 2017. 01 Version.

root@kali:~/lmg/volatility# uname -a
Linux kali 4.9.0-kali3-amd64 #1 SMP Debian 4.9.18-1kali1 (2017-04-04) x86_64 GNU/Linux

Unfortunatley, 0x9fb9bb000000 does not work.. But in the same condition with other computer, install same distro, did same work, that works well with only one row result. I don't know why it works ... very interesting. thanks!!

[atcuno]

Hello,

Can you please git pull to the latest version of Volatility and let me know if you have any further issues with analysis or profile building on 4.9+ kernels?

[cpuu]

@atcuno
I deleted all files and git clone new version. I found that 'DW_AT_byte_size' problem was solved. but need to handle kaslr_shift problem yet.

[atcuno]

Okay, can you please copy/paste your command line input/output when running Volatility against the 4.9 sample with the new profile installed?

Thanks, Andrew (@attrc)

On 05/08/2017 09:31 AM, cpuu wrote:

@atcuno
I deleted all files and git clone new version. 'DW_AT_byte_size' problem was solved. but need to handle kaslr_shift problem yet.

[cpuu]

I like to use lmg(automated script). There are config file with options "--conf-file=../capture/kali-2017-05-10_01.04.02/volatilityrc"

root@kali:~/lmg/capture/kali-2017-05-10_01.04.02# cat volatilityrc 
[DEFAULT]
PLUGINS=/root/lmg/capture/kali-2017-05-10_01.04.02
PROFILE=Linuxkali-2017-05-10_01_04_02-profilex64
LOCATION=file:////root/lmg/capture/kali-2017-05-10_01.04.02/kali-2017-05-10_01.04.02-memory.lime

I used linux_pslist and linux_aslr_shift, but error blow

root@kali:~/lmg/volatility# python vol.py --conf-file=../capture/kali-2017-05-10_01.04.02/volatilityrc linux_aslr_shift
Volatility Foundation Volatility Framework 2.6
Virtual Shift Address Physical Shift Address
--------------------- ----------------------
No suitable address space mapping found
Tried to open image as:
 MachOAddressSpace: mac: need base
 LimeAddressSpace: lime: need base
 WindowsHiberFileSpace32: No base Address Space
 WindowsCrashDumpSpace64BitMap: No base Address Space
 WindowsCrashDumpSpace64: No base Address Space
 HPAKAddressSpace: No base Address Space
 VirtualBoxCoreDumpElf64: No base Address Space
 VMWareMetaAddressSpace: No base Address Space
 QemuCoreDumpElf: No base Address Space
 VMWareAddressSpace: No base Address Space
 WindowsCrashDumpSpace32: No base Address Space
 Win10AMD64PagedMemory: No base Address Space
 WindowsAMD64PagedMemory: No base Address Space
 LinuxAMD64PagedMemory: No base Address Space
 AMD64PagedMemory: No base Address Space
 IA32PagedMemoryPae: No base Address Space
 IA32PagedMemory: No base Address Space
 OSXPmemELF: No base Address Space
 MachOAddressSpace: MachO Header signature invalid
 MachOAddressSpace: MachO Header signature invalid
 LimeAddressSpace: Invalid Lime header signature
 WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
 WindowsCrashDumpSpace64BitMap: Header signature invalid
 WindowsCrashDumpSpace64: Header signature invalid
 HPAKAddressSpace: Invalid magic found
 VirtualBoxCoreDumpElf64: ELF Header signature invalid
 VMWareMetaAddressSpace: VMware metadata file is not available
 QemuCoreDumpElf: ELF Header signature invalid
 VMWareAddressSpace: Invalid VMware signature: 0x905c689
 WindowsCrashDumpSpace32: Header signature invalid
 Win10AMD64PagedMemory: Incompatible profile Linuxkali-2017-05-10_01_04_02-profilex64 selected
 WindowsAMD64PagedMemory: Incompatible profile Linuxkali-2017-05-10_01_04_02-profilex64 selected
 LinuxAMD64PagedMemory: Failed valid Address Space check
 AMD64PagedMemory: Failed valid Address Space check
 IA32PagedMemoryPae: Incompatible profile Linuxkali-2017-05-10_01_04_02-profilex64 selected
 IA32PagedMemory: Incompatible profile Linuxkali-2017-05-10_01_04_02-profilex64 selected
 OSXPmemELF: ELF Header signature invalid
 FileAddressSpace: Must be first Address Space
 ArmAddressSpace: Failed valid Address Space check

So I download bneuburg's kaslr_shift.py , and use it.

root@kali:~/lmg/volatility# python vol.py --conf-file=../capture/kali-2017-05-10_01.04.02/volatilityrc linux_kaslr_shift
Volatility Foundation Volatility Framework 2.6
Traceback (most recent call last):
  File "vol.py", line 192, in <module>
    main()
  File "vol.py", line 183, in main
    command.execute()
  File "/root/lmg/volatility/volatility/plugins/linux/common.py", line 67, in execute
    commands.Command.execute(self, *args, **kwargs)
  File "/root/lmg/volatility/volatility/commands.py", line 147, in execute
    func(outfd, data)
  File "/root/lmg/volatility/volatility/commands.py", line 282, in render_text
    config = self._config), data)
  File "/root/lmg/volatility/volatility/commands.py", line 276, in _render
    renderer.render(outfd, output)
  File "/root/lmg/volatility/volatility/renderers/text.py", line 115, in render
    grid_depth = grid.visit(None, lambda x, y: max(y, grid.path_depth(x)), 0)
  File "/root/lmg/volatility/volatility/renderers/__init__.py", line 248, in visit
    self.populate()
  File "/root/lmg/volatility/volatility/renderers/__init__.py", line 149, in populate
    for (level, item) in self._generator:
  File "/root/lmg/volatility/volatility/plugins/linux/kaslr_shift.py", line 189, in generator
    for dtb, virtualshift, physicalshift in data:
  File "/root/lmg/volatility/volatility/plugins/linux/kaslr_shift.py", line 115, in calculate
    shift = profile.shift_address
AttributeError: 'Linuxkali-2017-05-10_01_04_02-profilex64' object has no attribute 'shift_address'

Oh... Did Something conflict?

[cpuu]

Interesting paper here. Derandomizing Kernel Address Space Layout for Memory Introspection and Forensics, CODASPY16

[Theory5]

@cpuu Did you ever figure out what was conflicting? I'm in the same situation, same frustrations, I tried the same linux_kaslr_shift.py script, and I get the same damn error, except it's using my own profile I built called LinuxUbuntu1804guix64, based off of 4.18.0-18-generic kernel.

I don't understand the connection between the paper you referenced before it was closed by @atcuno , and the error above, insofar as using both to produce a working memory dump file or fix linux_kaslr_shift.py.

I found some code modifications outside of that linux_kaslr_shift.py file @bneuburg created in his version of volatility's code, but I don't know if making those changes to my current 2.6 will work, given the length of time between his changes for the common files like linux.py and the current build I pulled for Volatility, and it appears that the code was never actually merged with the main volatility branch due to conflicts. I'm guessing with my understanding and luck, those changes will break my Volatility further.

I'm not really the best when it comes to this stuff, but I am pulling my hair out, because I followed the wiki exactly, build a new profile, and use the virtualbox dump methods (both) for the same exact Ubuntu kernel, and yet nothing works.

I even followed this guy (err, sorry, this girl):

https://www.andreafortuna.org/2017/06/23/how-to-extract-a-ram-dump-from-a-running-virtualbox-machine/

Who seemed to have a manual method and a quick script to move the address space. Nope, didn't work.

If necessary I'll open a new issue, but I don't understand the previous issues on this same matter nor how some of them reached a conclusion (like this issue) so I need to figure that out first.

I've gone from trying to dynamically analyze a packed ELF that was messed with just enough so I couldn't unpack it with UPX, to down a rabbit hole trying to get Volatility to work for this (and it's taken hours just getting to this point from the wiki, googling, and following these issues).

Any ideas? Thanks in advance!

[pagabuc]

Hey Theory5, while I definitely agree that there should be more support for linux KASLR, a trick I often use when I need to do tests is to simply disable KASLR. The accepted answer here should work: https://unix.stackexchange.com/questions/397939/turning-off-kaslr-to-debug-linux-kernel-using-qemu-and-gdb

Good luck!

[Theory5]

@pagabuc wow, thanks for the prompt reply! Since I'm just pulling memory from my malware analysis vbox vm, this should be a snap to impliment!

I didn't even know this was an option! If this works, I'm going to rewrite part of the wiki concerning vbox memory analysis and see if they'll add details concerning KASLR!

Thanks a ton!

[Theory5]

@pagabuc IT WORKS! Jesus H. Tapdancing Christ, it works! I ran the GRUB changes recommended, rebooted my VM in Vbox, and I'm getting some good info.

Volatility first said "SLUB Unsupported" not sure what that is about, but it then reported psxview output properly, as far as I understand it. I will start writing this up to request a change in the wiki unless anything else gives me problems. I just have to put this to the test with my malware analysis!

[pagabuc]

@Theory5 I am glad it helped! I was in your exact same situation a couple of years ago - and ever since I always disable KASLR before running tests :)

For the "SLUB Unsupported" the problem is that the kernel has 3 different memory allocators (SLAB, SLUB and SLOB) and some volatility plugins use the allocator structures (kmem_cache and friends) to locate and extract kernel objects. The problem is that volatility support is limited to SLAB, while Debian-based distros ship kernels that uses SLUB - and AFAIK this is something you can only change before compiling the kernel.

Have fun!