TypeError raised by windows.pslist on elf64 memory dump

[koromodako]

First of all, do not hesitate to rename this issue. It's always hard to find something precise enough without being too long.

windows.pslist plugin is raising TypeError when analyzing a memory dump (elf64 format). This issue might be related to #97.

INFO     volatility.framework.automagic: Detected a windows category plugin
INFO     volatility.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 9  volatility.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
INFO     volatility.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility.framework.automagic: Running automagic: LayerStacker
DEBUG    volatility.framework: Importing module: volatility.framework.layers.lime
DEBUG    volatility.framework: Importing module: volatility.framework.layers.linear
DEBUG    volatility.framework: Importing module: volatility.framework.layers.resources
DEBUG    volatility.framework: Importing module: volatility.framework.layers.crash
DEBUG    volatility.framework: Importing module: volatility.framework.layers.registry
DEBUG    volatility.framework: Importing module: volatility.framework.layers.elf
DEBUG    volatility.framework: Importing module: volatility.framework.layers.intel
DEBUG    volatility.framework: Importing module: volatility.framework.layers.msf
DEBUG    volatility.framework: Importing module: volatility.framework.layers.segmented
DEBUG    volatility.framework: Importing module: volatility.framework.layers.vmware
DEBUG    volatility.framework: Importing module: volatility.framework.layers.physical
DEBUG    volatility.framework: Importing module: volatility.framework.layers.scanners.multiregexp
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
DEBUG    volatility.framework.automagic.windows: Self-referential pointer not in well-known location, moving to recent windows heuristic
INFO     volatility.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility.schemas: All validations will report success, even with malformed input
DEBUG    volatility.framework.automagic.windows: DTB was found at: 0x187000
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary.memory_layer
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary.memory_layer.base_layer
INFO     volatility.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility.schemas: All validations will report success, even with malformed input
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 9  volatility.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
DEBUG    volatility.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'Elf64Layer', 'FileLayer']
INFO     volatility.framework.automagic: Running automagic: WintelHelper
INFO     volatility.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - using KDBG structure for kernel offset
INFO     volatility.framework.symbols.windows.pdbconv: Download PDB file...
DEBUG    volatility.framework.symbols.windows.pdbconv: Attempting to retrieve http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/4EC39E6D760F4A26A5192B89B2E0158E1/ntkrnlmp.pd_
DEBUG    volatility.framework.symbols.windows.pdbconv: Failed with HTTP Error 404: Not Found
DEBUG    volatility.framework.symbols.windows.pdbconv: Attempting to retrieve http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/4EC39E6D760F4A26A5192B89B2E0158E1/ntkrnlmp.pdb
DEBUG    volatility.framework.symbols.windows.pdbconv: Successfully written to /tmp/tmp10bx0m1k.pdb/4EC39E6D760F4A26A5192B89B2E0158E1/
Level 9  volatility.framework.configuration.requirements: TypeError - SymbolTableRequirement only accepts string labels: None2E0158E1/ntkrnlmp.pdb
WARNING  volatility.framework.plugins: Automagic exception occured: TypeError: __new__() missing 4 required positional arguments: 'type_name', 'object_info', 'base_type', and 'choices'
Level 9  volatility.framework.plugins: Traceback (most recent call last):
  File "/home/user/volatility3/volatility/framework/automagic/__init__.py", line 129, in run
    automagic(context, config_path, requirement, progress_callback)
  File "/home/user/volatility3/volatility/framework/automagic/pdbscan.py", line 479, in __call__
    self.recurse_symbol_fulfiller(context, valid_kernels, progress_callback)
  File "/home/user/volatility3/volatility/framework/automagic/pdbscan.py", line 209, in recurse_symbol_fulfiller
    self.download_pdb_isf(kernel['GUID'], kernel['age'], kernel['pdb_name'], progress_callback)
  File "/home/user/volatility3/volatility/framework/automagic/pdbscan.py", line 253, in download_pdb_isf
    json_output = pdbconv.PdbReader(self.context, location, progress_callback).get_json()
  File "/home/user/volatility3/volatility/framework/symbols/windows/pdbconv.py", line 263, in __init__
    self._layer_name, self._context = self.load_pdb_layer(context, location)
  File "/home/user/volatility3/volatility/framework/symbols/windows/pdbconv.py", line 299, in load_pdb_layer
    new_context = context.clone()
  File "/home/user/volatility3/volatility/framework/interfaces/context.py", line 94, in clone
    return copy.deepcopy(self)
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.7/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/usr/lib/python3.7/copy.py", line 273, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.7/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.7/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/usr/lib/python3.7/copyreg.py", line 88, in __newobj__
    return cls.__new__(cls, *args)
TypeError: __new__() missing 4 required positional arguments: 'type_name', 'object_info', 'base_type', and 'choices'

Unsatisfied requirement plugins.PsList.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
  You have the correct symbol file for the requirement
  The symbol file is under the correct directory or zip file
  The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.nt_symbols']

[ikelos]

You're not the first person to have run into this, and I now have a sample where I can recreate the problem, so I'll look into it and see if I can resolve the issue. If the solution is a little complex then it may have to go through our pull request approval policy, but I'll keep this issued updated as I figure out what's what... 5:)

[ikelos]

Right, so it turned out this was an issue with the ELF layer I hastily threw together. The offsets for the segments were volatility objects (which therefore carried references to their context with them), and something about that meant it couldn't be marshalled/unmarshalled correctly (needed for context.clone() and also for pickling). So instead, the segments are now converted to plain old boring ints (rather than spangly volatility ints) and that should have resolved the problem. If you can update the elf64 branch and test it, then I should be able to close this one off... 5:)

[ikelos]

Ok, independent verification says that this is now resolved. Do feel free to reopen it if the issue persists for you though! 5:)

[ikelos]

Forgot to mention, fixed by commit 94dad4cfc19a3d90d252ca94b7ed83a6b1604878.