search

volatilityfoundation / volatility

An advanced memory forensics framework
http://volatilityfoundation.org/
GNU General Public License v2.0
7.39k stars 1.29k forks source link

KeyError: 'DW_AT_byte_size' error on Ubuntu-Linux 4.15.0_48-generic system using Python 2.7.12 #638

Open dmwharris opened 5 years ago

dmwharris commented 5 years ago

Hello, after creating a volatility profile for an Ubuntu-Linux 4.15.0_48-generic system using version 2.6 and running it against a LiME sample created with

insmod lime-4.15.0-48-generic.ko "path=/home/developer/lime-4.15.0-48-generic.lime format=lime"

I get the following error:

root@nuc2:/home/developer/Downloads/volatility-master# python vol.py --profile=LinuxUbuntu4_15_0-48-genericx64 -f /home/developer/lime-4.15.0-48-generic.lime modules Volatility Foundation Volatility Framework 2.6 Traceback (most recent call last): File "vol.py", line 192, in main() File "vol.py", line 183, in main command.execute() File "/home/developer/Downloads/volatility-master/volatility/commands.py", line 116, in execute if not self.is_valid_profile(profs[self._config.PROFILE]()): File "/home/developer/Downloads/volatility-master/volatility/plugins/overlays/linux/linux.py", line 216, in init obj.Profile.init(self, *args, kwargs) File "/home/developer/Downloads/volatility-master/volatility/obj.py", line 862, in init self.reset() File "/home/developer/Downloads/volatility-master/volatility/plugins/overlays/linux/linux.py", line 227, in reset self.load_vtypes() File "/home/developer/Downloads/volatility-master/volatility/plugins/overlays/linux/linux.py", line 264, in load_vtypes vtypesvar = dwarf.DWARFParser(dwarfdata).finalize() File "/home/developer/Downloads/volatility-master/volatility/dwarf.py", line 71, in init self.feed_line(line) File "/home/developer/Downloads/volatility-master/volatility/dwarf.py", line 162, in feed_line self.process_statement(parsed) #pylint: disable-msg=W0142 File "/home/developer/Downloads/volatility-master/volatility/dwarf.py", line 204, in process_statement self.vtypes[name] = [ int(data['DW_AT_byte_size'], self.base), {} ] KeyError: 'DW_AT_byte_size'

Python version is 2.7.12

I was using the dwarfdump associated with Ubuntu 16.01 xenial, which as 20120410-2+deb7u2build0.16.04.1.

Is this because version 2.6 doesn't support Linux 4.15.0-48-generic? Or should I use a later version of dwarfdump?

pagabuc commented 5 years ago

Hey @dmwharris, can you please share the profile? And yes, I suggest to try with another version of dwarfdump!

shehreyarahmedkohati commented 5 years ago

Hi also getting some what same error, I am using Volatility 2.6 to analyze a memory image from a Ubuntu 16.04 server captured using and in Lime format. I am getting the below error persistently. I have made a several profiles for the OS using the System map details. Error: sudo python vol.py -f "ram.lime" --profile=LinuxUbuntu1604-36x64 linux_pslist Volatility Foundation Volatility Framework 2.6 Failed to import volatility.plugins.registry.shutdown (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.getservicesids (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.timeliner (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.malware.apihooks (NameError: name 'distorm3' is not defined) Failed to import volatility.plugins.malware.servicediff (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.userassist (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.getsids (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.shellbags (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.evtlogs (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.tcaudit (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.shimcache (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.dumpregistry (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.lsadump (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.malware.threads (NameError: name 'distorm3' is not defined) Failed to import volatility.plugins.mac.apihooks_kernel (ImportError: No module named distorm3) Failed to import volatility.plugins.registry.amcache (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.mac.check_syscall_shadow (ImportError: No module named distorm3) Failed to import volatility.plugins.malware.svcscan (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.auditpol (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.ssdt (NameError: name 'distorm3' is not defined) Failed to import volatility.plugins.registry.registryapi (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.mac.apihooks (ImportError: No module named distorm3) Failed to import volatility.plugins.envars (ImportError: No module named Crypto.Hash) Traceback (most recent call last): File "vol.py", line 192, in main() File "vol.py", line 183, in main command.execute() File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/plugins/linux/common.py", line 64, in execute commands.Command.execute(self, args, kwargs) File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/commands.py", line 116, in execute if not self.is_valid_profile(profs[self._config.PROFILE]()): File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/plugins/overlays/linux/linux.py", line 216, in init obj.Profile.init(self, *args, kwargs) File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/obj.py", line 862, in init self.reset() File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/plugins/overlays/linux/linux.py", line 227, in reset self.load_vtypes() File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/plugins/overlays/linux/linux.py", line 264, in load_vtypes vtypesvar = dwarf.DWARFParser(dwarfdata).finalize() File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/dwarf.py", line 71, in init self.feed_line(line) File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/dwarf.py", line 162, in feed_line self.process_statement(parsed) #pylint: disable-msg=W0142 File "/home/digit/Downloads/lmg-master/LiME-1.9/src/volatility-master/volatility/dwarf.py", line 204, in process_statement self.vtypes[name] = [ int(data['DW_AT_byte_size'], self.base), {} ] KeyError: 'DW_AT_byte_size'

Any help would be appreciated!

atcuno commented 5 years ago

Can you please confirm that you are using the latest master checkout of Volatility from here (github)? Line 204 does not match your backtrace: https://github.com/volatilityfoundation/volatility/blob/master/volatility/dwarf.py#L204

atcuno commented 4 years ago

@shehreyarahmedkohati (and others), please git pull and try to run Volatility with your profiles again. I made an update to dwarf.py to hopefully address everyone at once:

https://github.com/volatilityfoundation/volatility/commit/7b3f52b66935879c625f72bfb3430d741fefc24b

atcuno commented 4 years ago

Closed this by accident.

atcuno commented 4 years ago

@olifre please git pull again and re-try

olifre commented 4 years ago

@atcuno That's interesting! Now it runs for ~5 minutes and uses some GB of memory, so it seems the dwarf parsing went fine. Then, I end up at a (Pdb) prompt. Typing w, I get:

Volatility Foundation Volatility Framework 2.6.1
> /somewhere/volatility/plugins/overlays/linux/linux.py(262)_merge_anonymous_members()
-> raise exceptions.VolatilityException("Inconsistent linux profile - unable to look up " + str(e))
(Pdb) w
  /somewhere/vol.py(192)<module>()
-> main()
  /somewhere/vol.py(183)main()
-> command.execute()
  /somewhere/volatility/plugins/linux/common.py(67)execute()
-> commands.Command.execute(self, *args, **kwargs)
  /somewhere/volatility/commands.py(116)execute()
-> if not self.is_valid_profile(profs[self._config.PROFILE]()):
  /somewhere/volatility/plugins/overlays/linux/linux.py(218)__init__()
-> obj.Profile.__init__(self, *args, **kwargs)
  /somewhere/volatility/obj.py(862)__init__()
-> self.reset()
  /somewhere/volatility/plugins/overlays/linux/linux.py(232)reset()
-> self.load_vtypes()
  /somewhere/volatility/plugins/overlays/linux/linux.py(270)load_vtypes()
-> self._merge_anonymous_members(vtypesvar)
> /somewhere/volatility/plugins/overlays/linux/linux.py(262)_merge_anonymous_members()
-> raise exceptions.VolatilityException("Inconsistent linux profile - unable to look up " + str(e))
olifre commented 4 years ago

@atcuno Do I interpret correctly that this (likely) means this issue is solved and we now have https://github.com/volatilityfoundation/volatility3/issues/222 reproduced with Volatility 2?

atcuno commented 4 years ago

It is not solved. I am working on it, but getting those new types to parse is being difficult.

olifre commented 4 years ago

Thanks! No worries, I was just interpreting this as being back to the original issue now (for which we are still unsure whether the dump has some issues), but since that's not the case, for sure take your time implementing those new types.

lain3d commented 4 years ago

is there a minimum commit id where this isn't a problem

lain3d commented 4 years ago

any news on this?

canDry commented 4 years ago

I'm assuming this is the same issue...

python vol.py --info Volatility Foundation Volatility Framework 2.6.1 Failed to import volatility.plugins.registry.shutdown (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.getservicesids (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.timeliner (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.malware.apihooks (NameError: name 'distorm3' is not defined) Failed to import volatility.plugins.malware.servicediff (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.userassist (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.getsids (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.shellbags (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.evtlogs (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.tcaudit (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.dumpregistry (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.registry.lsadump (ImportError: No module named Crypto.Hash) Failed to import volatility.plugins.malware.threads (NameError: name 'distorm3' is not defined) Failed to import volatility.plugins.mac.apihooks_kernel (ImportError: No module named distorm3)

Ubuntu 18LTS... Python 2.7.17

Can someone plz confirm whether this is the same (or different) issue?

Update: hadn't installed them ie:

apt install python-crypto python-distorm3

Works now.

vincentroberge commented 2 years ago

On a Ubuntu 22.04 image, I get the following error. It seems to be related to this thread. Any help would be appreciated. I use the latest version of Volatility 2.6.1 just pulled from github yesterday. Thank you.

$ vol.py -f ubuntu.vmem --profile=Linuxubuntu-22.04-desktop-amd64_5.15.0-33-genericx64 linux_pslist
Volatility Foundation Volatility Framework 2.6.1
Traceback (most recent call last):
  File "/usr/local/bin/vol.py", line 192, in <module>
    main()
  File "/usr/local/bin/vol.py", line 183, in main
    command.execute()
  File "/usr/local/lib/python2.7/dist-packages/volatility/plugins/linux/common.py", line 67, in execute
    commands.Command.execute(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/volatility/commands.py", line 116, in execute
    if not self.is_valid_profile(profs[self._config.PROFILE]()):
  File "/usr/local/lib/python2.7/dist-packages/volatility/plugins/overlays/linux/linux.py", line 218, in __init__
    obj.Profile.__init__(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/volatility/obj.py", line 862, in __init__
    self.reset()
  File "/usr/local/lib/python2.7/dist-packages/volatility/plugins/overlays/linux/linux.py", line 232, in reset
    self.load_vtypes()
  File "/usr/local/lib/python2.7/dist-packages/volatility/plugins/overlays/linux/linux.py", line 269, in load_vtypes
    vtypesvar = dwarf.DWARFParser(dwarfdata).finalize()
  File "/usr/local/lib/python2.7/dist-packages/volatility/dwarf.py", line 72, in __init__
    self.feed_line(line)
  File "/usr/local/lib/python2.7/dist-packages/volatility/dwarf.py", line 161, in feed_line
    self.process_variable(parsed['data'])
  File "/usr/local/lib/python2.7/dist-packages/volatility/dwarf.py", line 330, in process_variable
    data['DW_AT_decl_file'].split()[1], data['DW_AT_type']))
IndexError: list index out of range
Gathub22 commented 6 months ago

Just want to leave here there is a pending PR with a patch that adds DWARFv5 and fixes the same KeyError. The patch works for me. Maybe this could help to you :)