volatilityfoundation / volatility3

Volatility 3.0 development
http://volatilityfoundation.org/
Other
2.68k stars 458 forks source link

Unsatisfied requirement plugins.Info.kernel.symbol_table_name #929

Closed chratschan closed 1 year ago

chratschan commented 1 year ago

Describe the bug I'm unable to use volatility in any sense because it says the symbol table requirements were not fulfilled. I have the symbol tables packs supplied by volatility and they're in the symbols folder but it's still not allowing me to run any plugin such as windows.info

Context Volatility Version: Volatility 3 Framework 2.4.1 Operating System: Windows 11 Python Version: Python 3.10.10 Suspected Operating System: Windows 10 Command: python3 vol.py -vvvvv -f C:\Users\ChristianP\Documents\Cybersecurity\memdump\LOGAN-J02QNY2-0-20230328-160641.dmp windows.info

To Reproduce Steps to reproduce the behavior:

  1. Use pip3 to install the requirements, place the symbols tables in the symbols folder.
  2. Use command 'python3 vol.py -vvvvv -f C:\Users\user\Documents\Cybersecurity\memdump\pc.dmp windows.info'
  3. See error

Expected behavior A clear and concise description of what you expected to happen.

Example output

Volatility 3 Framework 2.4.1
INFO     volatility3.cli: Volatility plugins path: ['C:\\Users\\user\\volatility3\\volatility3\\plugins', 'C:\\Users\\user\\volatility3\\volatility3\\framework\\plugins']
INFO     volatility3.cli: Volatility symbols path: ['C:\\Users\\user\\volatility3\\volatility3\\symbols', 'C:\\Users\\user\\volatility3\\volatility3\\framework\\symbols']
Level 7  volatility3.cli: Cache directory used: C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\Roaming\volatility3
INFO     volatility3.framework.automagic: Detected a windows category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 7  volatility3.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, VolatilityHandler, JarHandler, OfflineHandler, LeechCoreHandler
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 8  volatility3.framework.automagic.stacker: Stacked WindowsCrashDump64Layer using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsIntelStacker
DEBUG    volatility3.framework.automagic.windows: Detecting Self-referential pointer for recent windows
DEBUG    volatility3.framework.automagic.windows: DtbSelfRef64bit test succeeded at 0x1ad000
DEBUG    volatility3.framework.automagic.windows: DTB was found at: 0x1ad000
Level 8  volatility3.framework.automagic.stacker: Stacked IntelLayer using WindowsIntelStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name.memory_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Info.kernel.layer_name.memory_layer.base_layer
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Info
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'WindowsCrashDump64Layer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility3.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Setting kernel_virtual_offset to 0xf80432a00000
INFO     volatility3.framework.symbols.windows.pdbconv: Download PDB file...
DEBUG    volatility3.framework.symbols.windows.pdbconv: Attempting to retrieve http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/3789767E34B7A48A3FC80CE12DE18E651/ntkrnlmp.pdb
DEBUG    volatility3.framework.layers.resources: Caching file at: C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\Roaming\volatility3\data_b60ed38c5d553e589a865409931d2129817fad40b88f89449cb1e3c0531797221a1580ecaa376e3b7f76d54ce0ad7ddd1fe7fee6d7639902e55c085fea75f02a.cache
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder
INFO     volatility3.framework.automagic: Running automagic: KernelModule
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.kernel.symbol_table_name
WARNING  volatility3.framework.plugins: Automagic exception occurred: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Roaming\\volatility3\\data_b60ed38c5d553e589a865409931d2129817fad40b88f89449cb1e3c0531797221a1580ecaa376e3b7f76d54ce0ad7ddd1fe7fee6d7639902e55c085fea75f02a.cache'
Level 9  volatility3.framework.plugins: Traceback (most recent call last):
  File "C:\Users\user\volatility3\volatility3\framework\automagic\__init__.py", line 138, in run
    automagic(context, config_path, requirement, progress_callback)
  File "C:\Users\user\volatility3\volatility3\framework\automagic\pdbscan.py", line 448, in __call__
    self.recurse_symbol_fulfiller(
  File "C:\Users\user\volatility3\volatility3\framework\automagic\pdbscan.py", line 123, in recurse_symbol_fulfiller
    PDBUtility.load_windows_symbol_table(
  File "C:\Users\user\volatility3\volatility3\framework\symbols\windows\pdbutil.py", line 113, in load_windows_symbol_table
    cls.download_pdb_isf(
  File "C:\Users\user\volatility3\volatility3\framework\symbols\windows\pdbutil.py", line 262, in download_pdb_isf
    filename = pdbconv.PdbRetreiver().retreive_pdb(
  File "C:\Users\user\volatility3\volatility3\framework\symbols\windows\pdbconv.py", line 960, in retreive_pdb
    with resources.ResourceAccessor(progress_callback).open(
  File "C:\Users\user\volatility3\volatility3\framework\layers\resources.py", line 182, in open
    with open(temp_filename, "wb") as cache_file:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Roaming\\volatility3\\data_b60ed38c5d553e589a865409931d2129817fad40b88f89449cb1e3c0531797221a1580ecaa376e3b7f76d54ce0ad7ddd1fe7fee6d7639902e55c085fea75f02a.cache'

Unsatisfied requirement plugins.Info.kernel.symbol_table_name:

A symbol table requirement was not fulfilled. Please verify that: The associated translation layer requirement was fulfilled You have the correct symbol file for the requirement The symbol file is under the correct directory or zip file The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.Info.kernel.symbol_table_name']

Text is preferred to screenshots for searching and to talk about specific parts of the output.

Additional information Add any other information about the problem here.

ikelos commented 1 year ago

Volatility downloads and caches symbol files it needs for versions of windows that aren't in the packs that we bundle (which to be fair are quite old now). It appears as though volatility caches the file, but that something causes the filesystem writes to fail (or later access not to work). In this case the file being written was in C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Roaming\\volatility3\\. Please could you verify that the temporary storage does allow files to written, or use pdbconv.py to download and generate the necessary JSON file separately and ensure it's installed in the symbol directory at the correct (more recent versions of volatility allow symbol files to be found anywhere under the symbol directory, but earlier versions of volatility require a specific directory structure).

chratschan commented 1 year ago

Volatility downloads and caches symbol files it needs for versions of windows that aren't in the packs that we bundle (which to be fair are quite old now). It appears as though volatility caches the file, but that something causes the filesystem writes to fail (or later access not to work). In this case the file being written was in C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Roaming\\volatility3\\. Please could you verify that the temporary storage does allow files to written, or use pdbconv.py to download and generate the necessary JSON file separately and ensure it's installed in the symbol directory at the correct (more recent versions of volatility allow symbol files to be found anywhere under the symbol directory, but earlier versions of volatility require a specific directory structure).

Sorry, I am a bit of a beginner. How would I verify that temp storage allows files to be written? I don't actually see anything at C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\Roaming\volatility3\, so maybe that indicates that it can't?

In addition I've been seeing this name "pdbconv.py" referenced when researching this issue, but I have no idea what it is or where to find it. I did install the requirements with pip. Does pdbconv.py come with volatility? I don't see it anywhere in the directory.

Sorry, I know I am quite the beginner to digital forensics so these questions may seem asinine.

ikelos commented 1 year ago

That's ok, but you might find the support slightly more timely on our slack server where there are more people around who may be able to answer your questions more promptly. 5:)

Yeah, if the directory doesn't exist, that might be a problem, I don't recall if volatility tries to create the temporary directory if it doesn't exist. pdbconv.py is a component of the framework but was also built to be possible to run standalone. You can find out more about how to run it in the documentation. It lives in volatility3/framework/symbols/windows/pdbconv.py, but it needs the rest of the framework to run which is why you need to do the trick with the path to get it to run properly. It wasn't really intended to be a major feature, just a useful little utility so it's handy to know about. It uses exactly the same mechanism as volatility though, so you might encounter the same error when trying to use it. I'd suggest giving that a go, you should be able to run it from the top level volatility3 directory with PYTHONPATH="." python volatility3/framework/symbols/windows/pdbconv.py -g 3789767E34B7A48A3FC80CE12DE18E651 -p ntkrnlmp.pdb (based on the output of your run).

I've attached a generated copy of the JSON to get you going, but it would be interesting to know if it also fails or succeeds so that we can try and fix this (and in case it happens again with other images, which is likely). You'll need to rename it from .json.txt to .json. I'm going to keep this open, because at the very least throwing a horrible failure rather than giving a good explanation of what's gone wrong is a bug in the program. Once you give pdbconv a try we can try to debug what's going wrong and figure out if we can solve it (also, after the pdbconv attempt, if you could try creating the directory C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Roaming\\volatility3\\ and see if that responds differently would also be helpful). 5:)

3789767E34B7A48A3FC80CE12DE18E65-1.json.txt

chratschan commented 1 year ago

Agh!!! It wasn't even an issue with the symbols packs or data not being able to be wrote (I think)

The issue was that I made the mistake of using Python from the windows store. I compiled from source and now it's working as expected. Thanks windows store, you silly inbred goofball of an app store. I really appreciate your help, and I joined your slack just in case : )

ikelos commented 1 year ago

No problem, I'm glad you got it resolved, and thanks for reporting back the solution! The path for the file did suggest python from the windows store, but I was hoping they hadn't tampered with it too much, apparently they changed it enough to break things... 5:S Anyway, glad you got it solved, see you on slack sometime! 5:)