search

volatilityfoundation / volatility3

Volatility 3.0 development
http://volatilityfoundation.org/
Other
2.73k stars 463 forks source link

The windows.dumpfiles.DumpFiles plugin cannot dump all the files I want to dump.Some files can be dumped, some files cannot be dumped #940

Closed or4gevi closed 2 months ago

or4gevi commented 1 year ago

Describe the bug A clear and concise description of what the bug is. vCenter suspended the VM. Downloaded the VMEM file (16gb) and attempted to use Volatility3. The windows.dumpfiles plugin cannot dump all the files I want to dump.Some files can be dumped, some files cannot be dumped

Context Volatility Version: 3 Suspected Operating System: Windows Server 2016 10.0.14393

To Reproduce Steps to reproduce the behavior:

Command: $ python3 vol.py -vvv -f xx.vmem windows.info

Is64Bit True
IsPAE   False
layer_name  0 WindowsIntel32e
memory_layer    1 VmwareLayer
base_layer  2 FileLayer
meta_layer  2 FileLayer
KdVersionBlock  0xf801526edcf8
Major/Minor 15.14393
MachineType 34404
KeNumberProcessors  1
SystemTime  2023-03-29 01:58:49
NtSystemRoot    C:\Windows
NtProductType   NtProductLanManNt
NtMajorVersion  10
NtMinorVersion  0INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input

PE MajorOperatingSystemVersion  10
PE MinorOperatingSystemVersion  0
PE Machine  34404
PE TimeDateStamp    Mon Oct  9 01:45:44 2017

$ python3 vol.py -vvv -f xx.vmem windows.filescan.FileScan |grep ntds.dit

INFO     volatility3.cli: Volatility plugins path: ['/home/volatility3-2.4.1/volatility3/plugins', '/home/volatility3-2.4.1/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/home/volatility3-2.4.1/volatility3/symbols', '/home/volatility3-2.4.1/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a windows category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.windows: Detecting Self-referential pointer for recent windows
DEBUG    volatility3.framework.automagic.windows: DtbSelfRef64bit test succeeded at 0x1aa000
DEBUG    volatility3.framework.automagic.windows: DTB was found at: 0x1aa000
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name.memory_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name.memory_layer.base_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.FileScan.kernel.layer_name.memory_layer.meta_layer
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.FileScan
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'VmwareLayer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility3.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.FileScan.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Setting kernel_virtual_offset to 0xf80152402000
DEBUG    volatility3.framework.symbols.windows.pdbutil: Using symbol library: ntkrnlmp.pdb/35B4FD549B8D4779BEEF22E3E2BF3984-1
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder    
INFO     volatility3.framework.automagic: Running automagic: KernelModule
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EPROCESS_QUOTA_BLOCK
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PAGEFAULT_HISTORY
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_ACCESS_STATE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_CPU_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NET_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NOTIFICATION_INFORMATION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PSP_STORAGE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ACTIVATION_CONTEXT_DATA
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_FLS_CALLBACK_INFO
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ASSEMBLY_STORAGE_MAP
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_WNF_SCOPE_MAP
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_SOFT_RESTART_CONTEXT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_STACK_CACHE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_PERFECT_HASH_FUNCTION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_HAL_PMC_COUNTERS
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_SCSI_REQUEST_BLOCK
0xd10ac0648430  \Windows\NTDS\ntds.dit  216
...

python3 vol.py -vvv -f xx.vmem windows.dumpfiles.DumpFiles --virtaddr 0xd10ac0648430

Volatility 3 Framework 2.4.1
INFO     volatility3.cli: Volatility plugins path: ['/home/volatility3-2.4.1/volatility3/plugins', '/home/volatility3-2.4.1/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/home/volatility3-2.4.1/volatility3/symbols', '/home/volatility3-2.4.1/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a windows category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles
Level 9  volatility3.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - physaddr requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - physaddr requirements only accept int type: None
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.windows: Detecting Self-referential pointer for recent windows
DEBUG    volatility3.framework.automagic.windows: DtbSelfRef64bit test succeeded at 0x1aa000
DEBUG    volatility3.framework.automagic.windows: DTB was found at: 0x1aa000
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name.memory_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name.memory_layer.base_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.DumpFiles.kernel.layer_name.memory_layer.meta_layer
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.DumpFiles
Level 9  volatility3.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - pid requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - physaddr requirements only accept int type: None
Level 9  volatility3.framework.interfaces.configuration: TypeError - physaddr requirements only accept int type: None
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'VmwareLayer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility3.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.DumpFiles.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Setting kernel_virtual_offset to 0xf80152402000
DEBUG    volatility3.framework.symbols.windows.pdbutil: Using symbol library: ntkrnlmp.pdb/35B4FD549B8D4779BEEF22E3E2BF3984-1
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder    
INFO     volatility3.framework.automagic: Running automagic: KernelModule

Cache   FileObject  FileName    Result
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EPROCESS_QUOTA_BLOCK
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PAGEFAULT_HISTORY
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_ACCESS_STATE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_CPU_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NET_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NOTIFICATION_INFORMATION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PSP_STORAGE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ACTIVATION_CONTEXT_DATA
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_FLS_CALLBACK_INFO
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ASSEMBLY_STORAGE_MAP
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_SCSI_REQUEST_BLOCK
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_WNF_SCOPE_MAP
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_SOFT_RESTART_CONTEXT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_STACK_CACHE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_PERFECT_HASH_FUNCTION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_HAL_PMC_COUNTERS
ikelos commented 1 year ago

Hi there, Volatility can only dump the files it can identify from the memory image provided. There's no guarantee that all files you requested will be present in memory, or that the structures needed to find them will be present. If there's a specific file that you expected dumpfiles to be able to extract and can show that it's accessible through the technique that dumpfiles uses to locate the files it extracts, then we can investigate that bug, but the output you provided makes no mention of files that couldn't be recovered (nor any that could), so it's unclear that volatility isn't operating as expected. If you could provide more information as to why you believe volatility should have been able to determine the contents of a specific file from memory then we can investigate further... 5:)

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 200 days with no activity.

stream1990 commented 11 months ago

Hi there, Volatility can only dump the files it can identify from the memory image provided. There's no guarantee that all files you requested will be present in memory, or that the structures needed to find them will be present. If there's a specific file that you expected dumpfiles to be able to extract and can show that it's accessible through the technique that dumpfiles uses to locate the files it extracts, then we can investigate that bug, but the output you provided makes no mention of files that couldn't be recovered (nor any that could), so it's unclear that volatility isn't operating as expected. If you could provide more information as to why you believe volatility should have been able to determine the contents of a specific file from memory then we can investigate further... 5:)

I've encountered the same issue. So, how can I proceed to export this file? After such a long time, is the domain database file still unrecognized? Do I need to send you an example of this file? Thank you.

ikelos commented 11 months ago

It's not about the type of file actually recorded there, it's about identifying the structure recording the information about the file. It turns out the technique volatility uses to find these files is different between the two plugins. The filescan plugin uses the poolscanner to hunt for entries, the dumpfiles looks for file handles to dump the contents of. As such, entries found in filescan may not be files the can be retrieved by dumpfiles. @iMHLv2 may be able to explain the difference a little better, and let us know if it would be possible to add a --dump option to the filescan plugin, but there's no guarantee the contents of the file are present in memory just because the entry is found in a pool.

github-actions[bot] commented 4 months ago

This issue is stale because it has been open for 200 days with no activity.

github-actions[bot] commented 2 months ago

This issue was closed because it has been inactive for 60 days since being marked as stale.