search

volatilityfoundation / volatility3

Volatility 3.0 development
http://volatilityfoundation.org/
Other
2.68k stars 460 forks source link

Changing subtype of `filp` returned by `Lsof` #922

Closed vobst closed 11 months ago

vobst commented 1 year ago

Describe the bug When calling the list_fds method of the lsof plugin the subtype of the returned filp behaves in (at least to me) unexpected ways. According to this snippet

file_type = symbol_table + constants.BANG + "file"

fds = objects.utility.array_of_pointers(
     fd_table, count=max_fds, subtype=file_type, context=context
)

for fd_num, filp in enumerate(fds):
     if filp != 0:
                # added by me ^^
                print(filp.dereference().vol.get("type_name"))
                filp = filp.dereference().cast(file_type)
                print(context.modules["kernel"].get_symbols_by_absolute_location(filp.f_op))
                # end
                full_path = LinuxUtilities.path_for_file(context, task, filp)

                yield fd_num, filp, full_path

It should always be symbol_table + constants.BANG + "file", i.e., symbol_table_name1!file. However, the actual output when running my plugin is

...
symbol_table_name1!file
['symbol_table_name1!bpf_map_fops']
symbol_table_name1!btf_type
...

i.e., there is a distinct point where it switches - the first time it hits an fd that represents a BPF map. In that case I use the returned file pointer's private_data to construct a BpfMap object, which internally does quite a lot of stuff (also involving BTF). However, I would not expect this to change anything in the fd iterator.

Context Volatility Version: a35afd4f343c10d7f8d1df2cb5eec8364c3dbd5a Operating System: Kali container on Arch host Python Version: 3.11.2 Suspected Operating System: self-compiled 6.0.12 Command:

To Reproduce Steps to reproduce the behavior:

  1. You can find the plugin that I was writing here
  2. An example memory image (+symbols) that triggers the behavior can be downloaded here (Note that the project contains a patch to hack around the issue and it gets applied if you use the container)

Expected behavior To me, the fact that something I do with the filp changes the subtype of the iterator for future items is unexpected. I am not familiar enough with your framework or Python to tell if this a mistake on my end, but if it is, I'd like to understand how I can change my code to avoid it.

Example output

vol -vvvvv -f /io/dumps/krie-3410c66d-26be0e1ef560.elf linux.bpf_proc
Volatility 3 Framework 2.4.2
INFO     volatility3.cli: Volatility plugins path: ['/opt/vol/volatility3/volatility3/plugins', '/opt/vol/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/opt/vol/volatility3/volatility3/symbols', '/opt/vol/volatility3/volatility3/framework/symbols']
Level 7  volatility3.cli: Cache directory used: /root/.cache/volatility3
INFO     volatility3.framework.automagic: Detected a linux category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 7  volatility3.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, VolatilityHandler, JarHandler, OfflineHandler, LeechCoreHandler
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Stacked Elf64Layer using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LinuxIntelStacker
DEBUG    volatility3.framework.automagic.symbol_cache: Duplicate entry for identifier b'Linux version 6.0.12 (user@26be0e1ef560) (Ubuntu clang version 14.0.0-1ubuntu1, Ubuntu LLD 14.0.0) #1 SMP PREEMPT_DYNAMIC Fri Mar 3 21:31:29 UTC 2023\n\x00': file:///opt/vol/volatility3/volatility3/symbols/linux/6.0.12_x86_64_dfrws_vol_ebpf-26be0e1ef560.json and file:///opt/vol/volatility3/volatility3/symbols/linux/c3404494065f55e73e9459dd23f8808f.isf.json
DEBUG    volatility3.framework.automagic.linux: Identified banner: b'Linux version 6.0.12 (user@26be0e1ef560) (Ubuntu clang version 14.0.0-1ubuntu1, Ubuntu LLD 14.0.0) #1 SMP PREEMPT_DYNAMIC Fri Mar 3 21:31:29 UTC 2023\n\x00'
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!assoc_array_ptr
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!io_notif_slot
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mtd_info
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!iopf_device_param
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!nf_ct_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mem_cgroup
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!sfp_bus
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!udp_tunnel_nic
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!pcpu_dstats
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!phylink
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!smc_hashinfo
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!reset_control
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!gpio_desc
DEBUG    volatility3.framework.automagic.linux: Linux ASLR shift values determined: physical 0 virtual 0
DEBUG    volatility3.framework.automagic.linux: DTB was found at: 0x300c000
Level 8  volatility3.framework.automagic.stacker: Stacked IntelLayer using LinuxIntelStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name.memory_layer
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.BpfPslist.kernel.layer_name.memory_layer.base_layer
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.BpfPslist
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'Elf64Layer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder
INFO     volatility3.framework.automagic: Running automagic: LinuxSymbolFinder
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.BpfPslist.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.symbol_cache: Duplicate entry for identifier b'Linux version 6.0.12 (user@26be0e1ef560) (Ubuntu clang version 14.0.0-1ubuntu1, Ubuntu LLD 14.0.0) #1 SMP PREEMPT_DYNAMIC Fri Mar 3 21:31:29 UTC 2023\n\x00': file:///opt/vol/volatility3/volatility3/symbols/linux/6.0.12_x86_64_dfrws_vol_ebpf-26be0e1ef560.json and file:///opt/vol/volatility3/volatility3/symbols/linux/c3404494065f55e73e9459dd23f8808f.isf.json
DEBUG    volatility3.framework.automagic.symbol_finder: Identified banner: b'Linux version 6.0.12 (user@26be0e1ef560) (Ubuntu clang version 14.0.0-1ubuntu1, Ubuntu LLD 14.0.0) #1 SMP PREEMPT_DYNAMIC Fri Mar 3 21:31:29 UTC 2023\n\x00'
DEBUG    volatility3.framework.automagic.symbol_finder: Using symbol library: file:///opt/vol/volatility3/volatility3/symbols/linux/6.0.12_x86_64_dfrws_vol_ebpf-26be0e1ef560.json
INFO     volatility3.framework.automagic: Running automagic: KernelModule

PID     COMM    PROGS   MAPS    LINKS
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!assoc_array_ptr
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!io_notif_slot
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mtd_info
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!iopf_device_param
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!nf_ct_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mem_cgroup
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!sfp_bus
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!udp_tunnel_nic
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!pcpu_dstats
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!phylink
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!smc_hashinfo
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!reset_control
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!gpio_desc
Level 9  volatility3.utility.common: Program does not have BTF info attached
Level 9  volatility3.utility.common: Program does not have BTF info attached
Level 9  volatility3.utility.common: Program does not have BTF info attached
Level 9  volatility3.utility.common: Program does not have BTF info attached
Level 9  volatility3.utility.common: Program does not have BTF info attached

1       systemd 6,7,8,9,10              Traceback (most recent call last):
  File "/opt/vol/volatility3/vol.py", line 10, in <module>
    volatility3.cli.main()
  File "/opt/vol/volatility3/volatility3/cli/__init__.py", line 797, in main
    CommandLine().run()
  File "/opt/vol/volatility3/volatility3/cli/__init__.py", line 445, in run
    renderers[args.renderer]().render(constructed.run())
  File "/opt/vol/volatility3/volatility3/cli/text_renderer.py", line 193, in render
    grid.populate(visitor, outfd)
  File "/opt/vol/volatility3/volatility3/framework/renderers/__init__.py", line 241, in populate
    for level, item in self._generator:
  File "/opt/vol/volatility3/volatility3/plugins/linux/bpf_proc.py", line 127, in _generator
    for task, progs, maps, links in self.list_bpf_procs(
  File "/opt/vol/volatility3/volatility3/plugins/linux/bpf_proc.py", line 96, in list_bpf_procs
    for pid, comm, _task, fd_fields in fds_generator:
  File "/opt/vol/volatility3/volatility3/framework/plugins/linux/lsof.py", line 69, in list_fds
    for fd_fields in fd_generator:
  File "/opt/vol/volatility3/volatility3/framework/symbols/linux/__init__.py", line 229, in files_descriptors_for_process
    full_path = LinuxUtilities.path_for_file(context, task, filp)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vol/volatility3/volatility3/framework/symbols/linux/__init__.py", line 172, in path_for_file
    dentry = filp.get_dentry()
             ^^^^^^^^^^^^^^^
  File "/opt/vol/volatility3/volatility3/framework/objects/__init__.py", line 453, in __getattr__
    return getattr(self.dereference(), attr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vol/volatility3/volatility3/framework/objects/__init__.py", line 968, in __getattr__
    raise AttributeError(
AttributeError: StructType has no attribute: symbol_table_name1!btf_type.get_dentry
github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 200 days with no activity.

github-actions[bot] commented 11 months ago

This issue was closed because it has been inactive for 60 days since being marked as stale.

ikelos commented 11 months ago

I'd like to keep this open because I do still want to get to the bottom of it...

ikelos commented 11 months ago

Hi there, thanks very much for the detailed report and I can only apologize that it's taken us this long to dig into it. It was on our radar, but we didn't communicate that and it didn't have the priority it should have given it was a core issue.

You were correct, there was an issue with the way that array_of_pointers was constructing its arrays. It was taking the default pointer type class and updating its subtype (which then affected all subsequent copies of that type, ie, all pointers) whereas it should have cloned it and used only that. I've merged in the appropriate fix as of commit 9f9afbf. Thank you again for bringing it to our attention and the detailed information you provided.