Symbols error

[th-ink]

Describe the bug When running windows.strings.Strings plugin, returns Unsatisfied requirement plugins.Strings.nt_symbols: Windows kernel symbols error. The file volatilty tries to open (ntkrnlmp.pdb/57489119968749168D61EA066CAE9589-1) seems to be missing from symbols library found at: https://downloads.volatilityfoundation.org/volatility3/symbols/windows.zip,

Context Volatility Version: 1.0.1 Operating System: win10 Python Version: 3.6.9 Suspected Operating System: Windows Server 2008 Command: python3 vol.py -f raw_image windows.strings.Strings --strings-file interesting_strings.txt

Returned error:

DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.symbols.windows.pdbutil: Using symbol library: ntkrnlmp.pdb/57489119968749168D61EA066CAE9589-1
WARNING  volatility3.framework.plugins: Automagic exception occurred: EOFError: Compressed file ended before the end-of-stream marker was reached

[ikelos]

Hiya,

So even if the symbol file weren't present, volatility would try to grab the appropriate file from the Microsoft PDB servers and rebuild it. The interesting bit of this is the last line: Compressed file ended before the end-of-stream marker was reached. This suggests that either the XZ file that contains the specific JSON generated on your system, or the windows.zip file, is incomplete (and will need removing before you can get it to work).

The windows.zip file we have on the site doesn't look to include it (we can't include them all because new ones come out all the time), but volatility3 should generate them for you. It therefore looks like the one it tried to generated failed midway through but after it wrote the start of the file. Please could you delete the volatility3/symbols/windows/ntkrnlmp.pdb/57489119968749168D61EA066CAE9589-1.json.xz file that's there, and try running volatility again? It should go through the process of downloading and processing the appropriate pdb file? I've verified that the processing succeeds without issue, so as long as there's an internet connection and it can finish the process, everything should work.

Please let us know so we know what to do with this issue. I'll leave it open until we hear back... 5:)

[th-ink]

Hey, thanks for the quick reply. It seems that deleting the file in question really solves the problem as Volatility now download the file succesfully. However, there is a new error now, that reads:

INFO     volatility3.framework.symbols.windows.pdbconv: Download PDB file...
DEBUG    volatility3.framework.symbols.windows.pdbconv: Attempting to retrieve http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/57489119968749168D61EA066CAE95891/ntkrnlmp.pdb
WARNING  volatility3.framework.plugins: Automagic exception occurred: volatility3.framework.exceptions.InvalidAddressException: Offset outside of the buffer boundaries

Unsatisfied requirement plugins.Strings.nt_symbols: Windows kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
        You have the correct symbol file for the requirement
        The symbol file is under the correct directory or zip file

[ikelos]

Hmmm, that's very strange. Would you be able to run it with -vvvvv so we can see the full exception where it occurs. It's not clear whether it happened during the attempt to create the PDB file, or some time after that...

[th-ink]

sure, here is the output:

INFO     volatility3.framework.symbols.windows.pdbconv: Download PDB file...
DEBUG    volatility3.framework.symbols.windows.pdbconv: Attempting to retrieve http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/57489119968749168D61EA066CAE95891/ntkrnlmp.pdb
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Strings.nt_symbols5891/ntkrnlmp.pdb
WARNING  volatility3.framework.plugins: Automagic exception occurred: volatility3.framework.exceptions.InvalidAddressException: Offset outside of the buffer boundaries
Level 9  volatility3.framework.plugins: Traceback (most recent call last):
  File "/home/jwatts/volatility3/volatility3/framework/automagic/__init__.py", line 131, in run
    automagic(context, config_path, requirement, progress_callback)
  File "/home/jwatts/volatility3/volatility3/framework/automagic/pdbscan.py", line 329, in __call__
    self.recurse_symbol_fulfiller(context, valid_kernel, progress_callback)
  File "/home/jwatts/volatility3/volatility3/framework/automagic/pdbscan.py", line 110, in recurse_symbol_fulfiller
    progress_callback = progress_callback)
  File "/home/jwatts/volatility3/volatility3/framework/symbols/windows/pdbutil.py", line 80, in load_windows_symbol_table
    cls.download_pdb_isf(context, guid.upper(), age, pdb_name, progress_callback)
  File "/home/jwatts/volatility3/volatility3/framework/symbols/windows/pdbutil.py", line 207, in download_pdb_isf
    json_output = pdbconv.PdbReader(context, location, pdb_name, progress_callback).get_json()
  File "/home/jwatts/volatility3/volatility3/framework/symbols/windows/pdbconv.py", line 267, in __init__
    self._layer_name, self._context = self.load_pdb_layer(context, location)
  File "/home/jwatts/volatility3/volatility3/framework/symbols/windows/pdbconv.py", line 317, in load_pdb_layer
    msf_layer.read_streams()
  File "/home/jwatts/volatility3/volatility3/framework/layers/msf.py", line 67, in read_streams
    [x for x in root_pages])
  File "/home/jwatts/volatility3/volatility3/framework/layers/msf.py", line 67, in <listcomp>
    [x for x in root_pages])
  File "/usr/lib/python3.6/_collections_abc.py", line 883, in __iter__
    v = self[i]
  File "/home/jwatts/volatility3/volatility3/framework/objects/__init__.py", line 617, in __getitem__
    result += [self.vol.subtype(context = self._context, object_info = object_info)]
  File "/home/jwatts/volatility3/volatility3/framework/objects/templates.py", line 72, in __call__
    return self.vol.object_class(context = context, object_info = object_info, **arguments)
  File "/home/jwatts/volatility3/volatility3/framework/objects/__init__.py", line 121, in __new__
    value = cls._unmarshall(context, data_format, object_info)
  File "/home/jwatts/volatility3/volatility3/framework/objects/__init__.py", line 143, in _unmarshall
    data = context.layers.read(object_info.layer_name, object_info.offset, data_format.length)
  File "/home/jwatts/volatility3/volatility3/framework/interfaces/layers.py", line 551, in read
    return self[layer].read(offset, length, pad)
  File "/home/jwatts/volatility3/volatility3/framework/layers/linear.py", line 47, in read
    output += [self._context.layers.read(layer, mapped_offset, mapped_length, pad)]
  File "/home/jwatts/volatility3/volatility3/framework/interfaces/layers.py", line 551, in read
    return self[layer].read(offset, length, pad)
  File "/home/jwatts/volatility3/volatility3/framework/layers/linear.py", line 47, in read
    output += [self._context.layers.read(layer, mapped_offset, mapped_length, pad)]
  File "/home/jwatts/volatility3/volatility3/framework/interfaces/layers.py", line 551, in read
    return self[layer].read(offset, length, pad)
  File "/home/jwatts/volatility3/volatility3/framework/layers/physical.py", line 144, in read
    "Offset outside of the buffer boundaries")
volatility3.framework.exceptions.InvalidAddressException: Offset outside of the buffer boundaries

[ikelos]

Hmmmm, that suggests a bad PDB file, rather than anything to do with the memory image, oddly? Which is very strange since I download the same file and was able to process it fine? It might be worth removing ~/.cache/volatility3/data_*.cache files (although I think it might redownload the file anyway?). You could also try:

PYTHONPATH="." python volatility3/framework/symbols/windows/pdbconv.py -p ntkrnlmp.pdb -g 57489119968749168D61EA066CAE95891

which should reproduce the issue. Sorry for all the tests, but we are getting closer to figuring it out.... 5:)

[th-ink]

I tried both of your suggestions and it now works without a problem. Thank you very much!

[ikelos]

No problem! I'm going to close this issue off, but if it flares back up again, come back and see us and we can reopen this... 5;D

[blacowhait]

Hello! so i got the same error but in a different environment. My error is on Ubuntu 22 image with 5.15.0-58 kernel. Can you help configure how to solve it? i had tried your solution above but still got error :'D

[eve-mem]

Can you post the errors you get with the verbose flags please? (-vvvvv)

[FuckerLinda]

~/.cache/volatility3/data_*.cache

i love you. it works!!!!