Automagic windows heuristics fail on Win10x64_18362 memory sample

[gleeda]

Describe the bug For some reason on a Win10x64_18362 memory sample plugins are failing because "A symbol table requirement was not fulfilled" in spite of the fact that I have just pulled the updated windows.zip file.

Context Volatility Version: latest git clone Operating System: Win10x64_18362 Python Version: Python 3.7.3 Suspected Operating System:
Command:

$ python3 vol.py -f memory.raw  windows.pslist.PsList
Unsatisfied requirement plugins.PsList.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
    You have the correct symbol file for the requirement
    The symbol file is under the correct directory or zip file
    The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.nt_symbols']

This sample works fine with Volatility 2.6.1, btw.

[ikelos]

Hiya, could you rerun it with at least -vvvv please? That should tell us what the automagic's doing and why it's not succeeding...

[gleeda]

Ahh I forgot!

output.txt

[ikelos]

Ok, so it has stacked the intel layer, so it's during the hunt for the kernel offset that it's having difficulty. You said that volatility 2 ran it ok, could you take a look and find out the offset for the kernel module (ntoskrnl or whatever version it is that it's running) please? Also, it would probably be worth running it at with all 7 -vvvvvvv, so we can see what offset it chose for the DTB...

[gleeda]

For Volatility 2.6.1, the dtb value is 0x1ad002 in the attached output I see:

114 DEBUG volatility.framework.automagic.windows: DTB was found at: 0x1ad000

ntoskrnl info:

Offset(V)          Name                 Base                             Size File
------------------ -------------------- ------------------ ------------------ ----
0xffffad08cb248970 ntoskrnl.exe         0xfffff80578219000           0xab7000 \SystemRoot\system32\ntoskrnl.exe

[gleeda]

If I run volatility 2 w/the other dtb, it's still good:

$ python vol.py -f memory.raw --profile=Win10x64_18362 pslist --dtb=0x1ad000
Volatility Foundation Volatility Framework 2.6.1
Offset(V)          Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                          Exit                          
------------------ -------------------- ------ ------ ------ -------- ------ ------ ------------------------------ ------------------------------
0xffffad08cb270040 System                    4      0    133        0 ------      0 2020-05-05 20:25:55 UTC+0000                                 
0xffffad08cb2b0080 Registry                 88      4      4        0 ------      0 2020-05-05 20:25:47 UTC+0000                                 
0xffffad08ce19e400 smss.exe                332      4      2        0 ------      0 2020-05-05 20:25:55 UTC+0000                                 
0xffffad08ce87f140 csrss.exe               424    416     10        0      0      0 2020-05-05 20:26:01 UTC+0000                                 
0xffffad08cf27c140 wininit.exe             500    416      1        0      0      0 2020-05-05 20:26:01 UTC+0000     
[snip]

[ikelos]

Hmmmm, 0x1ad002 is pretty strange, the DTB (should) always start on a page boundary, which I suspect is why there's no problem using it for vol2. The kernel's also loaded really high, can you check the vtop on that address? We can try tweaking the algorithm to make it search for it for longer? The string search for KDB apparently failed, then searching for "SystemRoot\system32\nt" failed and then the last one tries at a fixed mapping (which tried several different possibilities all of which failed)...

[ikelos]

The other option is to create a json file as follows:

{
  "primary.class": "volatility.framework.layers.intel.WindowsIntel32e",
  "primary.kernel_virtual_offset": 272702373990400,
  "primary.memory_layer.class": "volatility.framework.layers.physical.FileLayer",
  "primary.memory_layer.location": "file:/path/to/memory.raw",
  "primary.page_map_offset": 1757186
}

And see whether that works? I might've got the kernel_virtual_offset wrong (I used the base, rather than the offset value, because I also expect that start of the kernel to be page aligned), but that's also worth a shot (to see if it can figure the rest of it out).

[ikelos]

You'll need to change the /path/to/memory.raw

[gleeda]

Seems to be correct:

Volatility Foundation Volatility Framework 2.6.1
Current context: System @ 0xffffad08cb270040, pid=4, ppid=0 DTB=0x1ad002
Python 2.7.16 (default, Feb 29 2020, 01:55:37) 
Type "copyright", "credits" or "license" for more information.

IPython 5.5.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: hex(addrspace().vtop(0xffffad08cb248970))
Out[1]: '0x7ce48970L'

In [2]: hex(addrspace().vtop(0xfffff80578219000))
Out[2]: '0x1f00000L'

In [3]: quit

$ xxd -s 0x1f00000 memory.raw|less
01f00000: 4d5a 9000 0300 0000 0400 0000 ffff 0000  MZ..............
01f00010: b800 0000 0000 0000 4000 0000 0000 0000  ........@.......
01f00020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
01f00030: 0000 0000 0000 0000 0000 0000 0801 0000  ................
01f00040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468  ........!..L.!Th
01f00050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
01f00060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
[snip]

$ xxd -s 0x7ce48970 memory.raw|less
[snip]
7ce48bd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
7ce48be0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
7ce48bf0: 0000 0000 0000 0000 6e00 7400 6f00 7300  ........n.t.o.s.
7ce48c00: 6b00 7200 6e00 6c00 2e00 6500 7800 6500  k.r.n.l...e.x.e.
7ce48c10: 0000 0000 0000 0000 0000 0000 0000 0000  ................
[snip]

[gleeda]

hrmmm that didn't work:

$ python3.7 vol.py  -c config.json windows.pslist.PsList
Volatility 3 Framework 1.0.0-beta.1
WARNING  volatility.framework.plugins: Automagic exception occurred: ValueError: Unable to run LayerStacker, single_location parameter not provided
WARNING  volatility.framework.plugins: Automagic exception occurred: ValueError: Unable to run LayerStacker, single_location parameter not provided

Unsatisfied requirement plugins.PsList.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
    You have the correct symbol file for the requirement
    The symbol file is under the correct directory or zip file
    The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.nt_symbols']

[ikelos]

So the question is, why doesn't 0xfffff80001f00000 map to 0x1f00000? And why didn't the KDBG or the module heuristics work?

[ikelos]

Oh, yeah, ok, because it's only a partial config, you still need to supply the -f parameter.

[gleeda]

weird... i thought that's what the "primary.memory_layer.location": "file:// parameter was for.

I see that the windows.zip file is from Oct 16 2019 could it be that the profiles just aren't updated?

I still get the original error even with the config file and specifying the location with -f

[ikelos]

So that file is which one the layer will be, and it should reconstruct that layer as necessary, but some automagic won't run without a single-location parameter (and the issue is, knowing which layer the automagic needs to populate, etc). The windows.zip file being old shouldn't matter because if it can identify the kernel PDB it will download and construct the correct JSON file for it.

[gleeda]

OK, I got it working. I took the ntoskrnl.exe from the memory sample using volatility 2:

$ python vol.py -f memory.raw --profile=Win10x64_18362  moddump -b 0xfffff80578219000 -D ~/Desktop/memory
Volatility Foundation Volatility Framework 2.6.1
Module Base        Module Name          Result
------------------ -------------------- ------
0xfffff80578219000 ntoskrnl.exe         OK: driver.fffff80578219000.sys

$ mv driver.fffff80578219000.sys ntoskrnl.exe 

Got the PDB file using pdbparser:

$ symchk.py -e ntoskrnl.exe 
Trying http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/BE3E0FF92C7A93433D4A950A037EF6561/ntkrnlmp.pd_
HTTP error 404
Trying http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/BE3E0FF92C7A93433D4A950A037EF6561/ntkrnlmp.pdb
[snip]

Created the profile:

$ PYTHONPATH="." python3 volatility/framework/symbols/windows/pdbconv.py --guid BE3E0FF92C7A93433D4A950A037EF6561 -f ntkrnlmp.pdb -o something.json

Then added that to the config.json file to force it to use that profile:

$ cat config.json 
{
  "primary.class": "volatility.framework.layers.intel.WindowsIntel32e",
  "primary.kernel_virtual_offset": 272702373990400,
  "primary.memory_layer.class": "volatility.framework.layers.physical.FileLayer",
  "primary.memory_layer.location": "file:///memory.raw",
  "primary.page_map_offset": 1757186,
  "nt_symbols.class": "volatility.framework.symbols.windows.WindowsKernelIntermedSymbols", "nt_symbols.isf_url":"file:///volatility3/something.json"
}

and finally got output:

$ python3.7 vol.py -c config.json -f memory.raw  windows.pslist.PsList 
Volatility 3 Framework 1.0.0-beta.1

PID PPID    ImageFileName   Offset(V)   Threads Handles SessionId   Wow64   CreateTime  ExitTime

4   0   System  0xad08cb270040  133 -   N/A False   2020-05-05 20:25:55.000000  N/A
88  4   Registry    0xad08cb2b0080  4   -   N/A False   2020-05-05 20:25:47.000000  N/A
332 4   smss.exe    0xad08ce19e400  2   -   N/A False   2020-05-05 20:25:55.000000  N/A
424 416 csrss.exe   0xad08ce87f140  10  -   0   False   2020-05-05 20:26:01.000000  N/A
500 416 wininit.exe 0xad08cf27c140  1   -   0   False   2020-05-05 20:26:01.000000  N/A
508 492 csrss.exe   0xad08cf24c0c0  11  -   1   False   2020-05-05 20:26:01.000000  N/A
564 492 winlogon.exe    0xad08cf2f70c0  2   -   1   False   2020-05-05 20:26:01.000000  N/A
[snip]

[ikelos]

Just to follow up on other conversations, after identifying the correct profile and forcing volatility to use it, everything worked just fine, so it's an instance where the windows automagic heuristics aren't successful. It will probably need a copy of the image to identify this further...

[ikelos]

Just to note, that there shouldn't be a need to add the file image when using a config.json, but it turned out that the various layers weren't being loaded before the reconstruction process was taking place. This has been fixed as of 94426ad3.

[ikelos]

Further information on this specific problem, the KDBG structure appears to be swapped out, so that means we've fallen back to the next heuristic (the mysterious module list that we use), so I'm investigating that a bit further...

[gleeda]

Just to note, that there shouldn't be a need to add the file image when using a config.json, but it turned out that the various layers weren't being loaded before the reconstruction process was taking place. This has been fixed as of 94426ad3.

This makes so much more sense to me. Thank you for fixing it :-D

[gleeda]

@ikelos there has been a change since we last visited this, that has broken my ability to get anything back from the memory sample now. I'll have to try to figure out what changed, but now the config file that I had above doesn't work anymore.

[ikelos]

Hmmm, ok, I'd probably need the output from vol.py -vvvvvvvv to help diagnose it. You could also try a git bisect to figure out the commit that causes the problem? The automagic system is a little delicate, so it may be there's something the automagic isn't now populating that it used to...

[gleeda]

At first it looks like the path isn't picked up from the config file now:

$ python3 vol.py -vvvvvvvv -c config.json windows.pslist.PsList
Volatility 3 Framework 2.0.0-beta.1
INFO     root        : Volatility plugins path: ['/Users/gleeda/Work/DEV/volatility3/volatility/plugins', '/Users/gleeda/Work/DEV/volatility3/volatility/framework/plugins']
INFO     root        : Volatility symbols path: ['/Users/gleeda/Work/DEV/volatility3/volatility/symbols', '/Users/gleeda/Work/DEV/volatility3/volatility/framework/symbols']
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/plugins, /Users/gleeda/Work/DEV/volatility3/volatility/framework/plugins
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.yarascan based on file: yarascan
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.svcscan based on file: windows/svcscan
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.hashdump based on file: windows/hashdump
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.cachedump based on file: windows/cachedump
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.vadyarascan based on file: windows/vadyarascan
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.callbacks based on file: windows/callbacks
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.lsadump based on file: windows/lsadump
INFO     volatility.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'pefile'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.verinfo based on file: windows/verinfo
INFO     root        : The following plugins could not be loaded (use -vv to see why): volatility.plugins.windows.cachedump, volatility.plugins.windows.callbacks, volatility.plugins.windows.hashdump, volatility.plugins.windows.lsadump, volatility.plugins.windows.svcscan, volatility.plugins.windows.vadyarascan, volatility.plugins.windows.verinfo, volatility.plugins.yarascan
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/automagic
Level 7  root        : Cache directory used: /Users/gleeda/.cache/volatility3
INFO     volatility.framework.automagic: Detected a windows category plugin
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
INFO     volatility.framework.automagic: Running automagic: ConstructionMagic
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary.memory_layer
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 7  volatility.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, JarHandler
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework.automagic.construct_layers: Construction Exception occurred: Unexpected config value found: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
INFO     volatility.framework.automagic: Running automagic: LayerStacker
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
INFO     volatility.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility.framework.automagic: Running automagic: WintelHelper
INFO     volatility.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - testing fixed base address
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80001f00000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80008d58000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80015e06000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80061d97000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80077e2a000
INFO     volatility.framework.automagic.pdbscan: No suitable kernels found during pdbscan
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
WARNING  volatility.framework.plugins: Automagic exception occurred: ValueError: Unable to run LayerStacker, single_location parameter not provided
Level 9  volatility.framework.plugins: Traceback (most recent call last):
  File "/Users/gleeda/Work/DEV/volatility3/volatility/framework/automagic/__init__.py", line 131, in run
    automagic(context, config_path, requirement, progress_callback)
  File "/Users/gleeda/Work/DEV/volatility3/volatility/framework/automagic/stacker.py", line 64, in __call__
    raise ValueError("Unable to run LayerStacker, single_location parameter not provided")
ValueError: Unable to run LayerStacker, single_location parameter not provided

Unsatisfied requirement plugins.PsList.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
    You have the correct symbol file for the requirement
    The symbol file is under the correct directory or zip file
    The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.nt_symbols']

[gleeda]

But it ultimately still fails, even if I give it the location:

$ python3 vol.py -vvvvvvvv -c config.json -f memory.raw windows.pslist.PsList
Volatility 3 Framework 2.0.0-beta.1
INFO     root        : Volatility plugins path: ['/Users/gleeda/Work/DEV/volatility3/volatility/plugins', '/Users/gleeda/Work/DEV/volatility3/volatility/framework/plugins']
INFO     root        : Volatility symbols path: ['/Users/gleeda/Work/DEV/volatility3/volatility/symbols', '/Users/gleeda/Work/DEV/volatility3/volatility/framework/symbols']
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/plugins, /Users/gleeda/Work/DEV/volatility3/volatility/framework/plugins
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.yarascan based on file: yarascan
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.svcscan based on file: windows/svcscan
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.hashdump based on file: windows/hashdump
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.cachedump based on file: windows/cachedump
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.vadyarascan based on file: windows/vadyarascan
INFO     volatility.plugins.yarascan: Python Yara module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'yara'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.callbacks based on file: windows/callbacks
DEBUG    volatility.framework: No module named 'Crypto'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.lsadump based on file: windows/lsadump
INFO     volatility.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility.framework: No module named 'pefile'
DEBUG    volatility.framework: Failed to import module volatility.plugins.windows.verinfo based on file: windows/verinfo
INFO     root        : The following plugins could not be loaded (use -vv to see why): volatility.plugins.windows.cachedump, volatility.plugins.windows.callbacks, volatility.plugins.windows.hashdump, volatility.plugins.windows.lsadump, volatility.plugins.windows.svcscan, volatility.plugins.windows.vadyarascan, volatility.plugins.windows.verinfo, volatility.plugins.yarascan
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/automagic
Level 7  root        : Cache directory used: /Users/gleeda/.cache/volatility3
INFO     volatility.framework.automagic: Detected a windows category plugin
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
INFO     volatility.framework.automagic: Running automagic: ConstructionMagic
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary.memory_layer
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 7  volatility.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, JarHandler
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework.automagic.construct_layers: Construction Exception occurred: Unexpected config value found: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
INFO     volatility.framework.automagic: Running automagic: LayerStacker
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 8  volatility.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility.framework.layers.elf: Exception: Bad magic 0x0 at file offset 0x0
Level 8  volatility.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using WindowsIntelStacker
DEBUG    volatility.framework.automagic.windows: Self-referential pointer not in well-known location, moving to recent windows heuristic
DEBUG    volatility.framework.automagic.windows: DTB was found at: 0x1ad000
Level 8  volatility.framework.automagic.stacker: Stacked IntelLayer using WindowsIntelStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility.framework.layers.elf: Exception: Offset 0x0 does not exist within the base layer
Level 8  volatility.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_shift requirements only accept int type: None
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.nt_symbols
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 9  volatility.framework.interfaces.configuration: TypeError - symbol_mask requirements only accept int type: None
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
Level 6  volatility.framework: Importing from the following paths: /Users/gleeda/Work/DEV/volatility3/volatility/framework/layers
DEBUG    volatility.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'FileLayer']
INFO     volatility.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility.framework.automagic: Running automagic: WintelHelper
INFO     volatility.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility.framework.automagic.pdbscan: Kernel base determination - testing fixed base address
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80001f00000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80008d58000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80015e06000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80061d97000
DEBUG    volatility.framework.automagic.pdbscan: Potential kernel_virtual_offset caused a page fault: 0xf80077e2a000
INFO     volatility.framework.automagic.pdbscan: No suitable kernels found during pdbscan
Level 9  volatility.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.nt_symbols

Unsatisfied requirement plugins.PsList.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
    You have the correct symbol file for the requirement
    The symbol file is under the correct directory or zip file
    The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.nt_symbols']

[gleeda]

This is the config file:

$ cat config.json 
{
    "primary.class": "volatility.framework.layers.intel.WindowsIntel32e",
    "primary.kernel_virtual_offset": 18446735301106831360,
    "primary.memory_layer.class": "volatility.framework.layers.physical.FileLayer",
    "primary.memory_layer.location": "file:///path/to/memory.raw",
    "primary.page_map_offset": 1757186,
    "nt_symbols.class": "volatility.framework.symbols.windows.WindowsKernelIntermedSymbols",    
    "nt_symbols.isf_url":"file:///Users/gleeda/Work/DEV/volatility3/new_symbols.json"
}

And these values work with this commit: 0c43beb42d3f1d7e3c7ac6ccb97c8f95c438042f

[ikelos]

So I think I've still go the memory image, and when I roll back to 0c43beb I still get no luck with windows.pslist? I've also tried in volshell to have it figure out the symbols in needs:

pdbutil.PDBUtility.get_guid_from_mz(self.context, 'primary', 18446735301106831360)

Which says that it can't read the dos header properly? Do you happen to have the new_symbols.json file, or the PDB's GUID/Age so I can try to bisect it myself please?

[ikelos]

(This is the md5 I've got for it: 1e415dbbdea9d46314247970052306d9)

[gleeda]

Yes that's it (1e415dbbdea9d46314247970052306d9). Did you use the same config file?

[gleeda]

Here's a copy of the ntoskrnl.exe, pdb, and new_symbols.json from that memory sample. items.zip

[gleeda]

I just confirmed that this works with that config file I copied into a comment earlier, and the new_symbols.json file:

$ git checkout 0c43beb42d3f1d7e3c7ac6ccb97c8f95c438042f
$ python3 vol.py -c config.json windows.pslist.PsList
Volatility 3 Framework 1.0.0-beta.1

PID PPID    ImageFileName   Offset(V)   Threads Handles SessionId   Wow64   CreateTime  ExitTime

4   0   System  0xad08cb270040  133 -   N/A False   2020-05-05 20:25:55.000000  N/A
88  4   Registry    0xad08cb2b0080  4   -   N/A False   2020-05-05 20:25:47.000000  N/A
332 4   smss.exe    0xad08ce19e400  2   -   N/A False   2020-05-05 20:25:55.000000  N/A
424 416 csrss.exe   0xad08ce87f140  10  -   0   False   2020-05-05 20:26:01.000000  N/A
500 416 wininit.exe 0xad08cf27c140  1   -   0   False   2020-05-05 20:26:01.000000  N/A
[snip]

[ikelos]

Ok, so this is actually due to the way we have to handle the symbol_shift on symbol tables for linux. You can add the line "nt_symbols.symbol_shift": 0 to the config, and that'll sort the problem. I'm also looking at not making the shift a requirement, but unfortunately the way linux calculates things, it need to know whether to look for it or not... 5:S

[ikelos]

For reference, here's the commit that caused the break: 085ccc47e0f43acf9414c05ddc8678f52e1f78eb

[gleeda]

Thank you, @ikelos that works! I'll close this out now

[gleeda]

Oh sorry, reopening, because the original issue is still there actually- that we needed to use the config file in the first place. Ugh, sorry about that.

[ikelos]

So, it turns out Microsoft started varying the fixed self-referential pointer is used. I've improved this a little on develop (we now scan from for self-referential pointers from 0x1e0 to 0x1ff) which seems to be the only range we've empirically seen values for in the past. Please could you verify if this gets further (ie, identifies the DTB)?

[ilay122]

@gleeda I think I know what the issue is. If you send me the dump I can verify it and make a pull request.

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 60 days since being marked as stale.