Volatility can't match the memory dump file (MacOS Monterey 12.6 build 21G115) to the symbol table created

[Lowengrube]

Hey there, so currently i'm facing problem in using Volatility 3 to analyse the ram dump file from MacOS Monterey 12.6 build 21G115, I had successfully created the symbol table for that OS version: ./dwarf2json mac --macho /Library/Developer/KDKs/KDK_12.6_21G115.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel --macho-symbols /Library/Developer/KDKs/KDK_12.6_21G115.kdk/System/Library/Kernels/kernel > 12.6.json

After that I copied 12.6.json to /path_to_volatility3/symbols/mac/ directory.

ISFinfo shows: python3 ./volatility3/vol.py isfinfo file:///Users/test/volatility3/volatility3/symbols/mac/allmacho.json Unknown 19 0 64681 392 b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'

Banners of image show: python3 ./volatility3/vol.py -f raw_dump_only_osxpmem.dump banners

0x18d60273  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x18d602d6  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x19301fc2  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f960273  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f9602d6  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1ff01fc2  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x47fae08e  Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x124601008 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x12548e2b9 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x12548e31c Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f5db09c6 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x30cd8be4e Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f0865046 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f2ee408e Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f3ae38c6 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f4af0d57 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x400e3f88e Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64

It looks like ok, but any operations with image (mac.bash/mac.lsof/mac.pstree and etc.) don't work: python3 ./volatility3/vol.py -vvvvvv -f raw_dump_only_osxpmem.dump mac.bash

Volatility 3 Framework 2.3.0
INFO     volatility3.cli: Volatility plugins path: ['/Users/test/volatility3/volatility3/plugins', '/Users/test/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/Users/test/volatility3/volatility3/symbols', '/Users/test/volatility3/volatility3/framework/symbols']
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/plugins, /Users/test/volatility3/volatility3/framework/plugins
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.yarascan based on file: /Users/test/volatility3/volatility3/framework/plugins/yarascan.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.svcscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/svcscan.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.hashdump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/hashdump.py
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.skeleton_key_check based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/skeleton_key_check.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.cachedump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/cachedump.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.vadyarascan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/vadyarascan.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.mftscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/mftscan.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/netscan.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netstat based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/netstat.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.lsadump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/lsadump.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.verinfo based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/verinfo.py
INFO     volatility3.cli: The following plugins could not be loaded (use -vv to see why): volatility3.plugins.windows.cachedump, volatility3.plugins.windows.hashdump, volatility3.plugins.windows.lsadump, volatility3.plugins.windows.mftscan, volatility3.plugins.windows.netscan, volatility3.plugins.windows.netstat, volatility3.plugins.windows.skeleton_key_check, volatility3.plugins.windows.svcscan, volatility3.plugins.windows.vadyarascan, volatility3.plugins.windows.verinfo, volatility3.plugins.yarascan
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/automagic
Level 7  volatility3.cli: Cache directory used: /Users/test/.cache/volatility3
INFO     volatility3.framework.automagic: Detected a mac category plugin
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 6  volatility3.framework.automagic.construct_layers: Construction Exception occurred: Unexpected config value found: None
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
Level 6  volatility3.framework.symbols.intermed: Searching for symbols in /Users/test/volatility3/volatility3/symbols, /Users/test/volatility3/volatility3/framework/symbols
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 7  volatility3.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, VolatilityHandler, JarHandler, OfflineHandler
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility3.framework.layers.elf: Exception: Bad magic 0x4034b50 at file offset 0x0
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using MacIntelStacker
DEBUG    volatility3.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 7  volatility3.framework.automagic.stacker: Exception during stacking: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object
Level 6  volatility3.framework.automagic.stacker: Traceback (most recent call last):

  File "/Users/test/volatility3/volatility3/framework/automagic/stacker.py", line 171, in stack_layer
    new_layer = stacker.stack(context, initial_layer, progress_callback)

  File "/Users/test/volatility3/volatility3/framework/automagic/mac.py", line 61, in stack
    table = mac.MacKernelIntermedSymbols(context = context,

  File "/Users/test/volatility3/volatility3/framework/symbols/mac/__init__.py", line 21, in __init__
    self.set_type_class('vm_map_object', extensions.vm_map_object)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 54, in _delegate_function
    return getattr(self._delegate, name)(*args, **kwargs)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 362, in set_type_class
    raise ValueError(f"Symbol type not in {self.name} SymbolTable: {name}")

ValueError: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object

Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: FileLayer
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['FileLayer']
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder  
INFO     volatility3.framework.automagic: Running automagic: MacSymbolFinder
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
INFO     volatility3.framework.automagic: Running automagic: KernelModule
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name

Unsatisfied requirement plugins.Bash.kernel.layer_name: 
Unsatisfied requirement plugins.Bash.kernel.symbol_table_name: 

A translation layer requirement was not fulfilled.  Please verify that:
    A file was provided to create this layer (by -f, --single-location or by config)
    The file exists and is readable
    The file is a valid memory image and was acquired cleanly

A symbol table requirement was not fulfilled.  Please verify that:
    The associated translation layer requirement was fulfilled
    You have the correct symbol file for the requirement
    The symbol file is under the correct directory or zip file
    The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.Bash.kernel.layer_name', 'plugins.Bash.kernel.symbol_table_name']

[ikelos]

So it did identify the correct banner, but we then immediately try to use a structure that wasn't present:

volatility3.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 7  volatility3.framework.automagic.stacker: Exception during stacking: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object
Level 6  volatility3.framework.automagic.stacker: Traceback (most recent call last):

  File "/Users/test/volatility3/volatility3/framework/automagic/stacker.py", line 171, in stack_layer
    new_layer = stacker.stack(context, initial_layer, progress_callback)

  File "/Users/test/volatility3/volatility3/framework/automagic/mac.py", line 61, in stack
    table = mac.MacKernelIntermedSymbols(context = context,

  File "/Users/test/volatility3/volatility3/framework/symbols/mac/__init__.py", line 21, in __init__
    self.set_type_class('vm_map_object', extensions.vm_map_object)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 54, in _delegate_function
    return getattr(self._delegate, name)(*args, **kwargs)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 362, in set_type_class
    raise ValueError(f"Symbol type not in {self.name} SymbolTable: {name}")

ValueError: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object

It's not clear if this structure's name has changed, or it was removed (or possibly if the symbol table was generated incorrectly, but it doesn't look like it). We'll need to do some investigation to figure out what the problem is. The vm_map_object type is one that we override with a custom handler, and it appears the custom handler isn't finding the original definition in the JSON. I've asked @atcuno to see whether the vm_map_object structure was renamed or removed from recent mac kernels...

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[ikelos]

Ping @atcuno, before this times out in a couple months, could you please check about the vm_map_object in the mac symbol tables?

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[ikelos]

@atcuno The stale ticket just got added, which means it's been a couple of months since I asked. Have you had a chance to check out what's going on with the symbol tables and the vm_map_object symbol?

[Abyss-W4tcher]

This structure was removed from the kernel, as well as many related vm_map ones.

The old/new versions are here :

• https://github.com/apple-open-source/macos/blob/12.0.1/xnu/osfmk/vm/vm_map.h#L140
• https://github.com/apple-open-source/macos/blob/14.3/xnu/osfmk/vm/vm_map.h#L239

Here is an article brieflly talking about it :

• https://saaramar.github.io/kmem_guard_t_blogpost/#bookkeeping

I was planning to update the framework, but It's gonna need more time and analysis to fix it. It mostly impacts mac.malfind.

[ikelos]

@Abyss-W4tcher thanks for the analysis! Perhaps @atcuno or @gcmoreira can help out now we know what it is?

[Abyss-W4tcher]

Hi, I will propose a patch in a PR soon, it's only in my fork right now.

I inform any dev here, to avoid potentially duplicating the same work 😃

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[ikelos]

Definitely not stale, just tricky to fix... 5:S Sorry I've been so slow on it, I'm finding it hard to get dedicated time to fix big issues like this. It is still on my list though.

[Abyss-W4tcher]

Hi, this should have been fixed in the two macOS PRs that were released a few months ago. However due to the current calendar I completely understand the lack of time to allocate on this subject 👍.

[ikelos]

Yeah, I thought I was producing a MacSymbolTable that was a facade for the two different mac tables and gave you the right table (with the right shift) based on which symbol you asked for? Did we still need that, I don't remember how we left it sadly... 5:S

[Abyss-W4tcher]

When you are ready, just comment on any of the PRs and I will provide you with a quick resume.

But basically, this issue is related to malfind (fixed by the "plugins" PR) and the "double module" aspect you mentioned relates to the automagic update PR supporting a new self-contained MACHO kernel in macOS.

[ikelos]

Ok thanks, as I say it hasn't slipped off my list but it needs me to find some time to sit and concentrate on it. Thanks for your understanding, it's really appreciated!