Volatility as a library issue with get_requirements with PsList plugin

[Dionkal]

I've been trying to use volatility as a library. I am following the official documentation and I'm in the Determine what configuration options a plugin requires section.

Here is my code so far:

import volatility3.plugins
import volatility3.symbols
from volatility3 import framework
from volatility3.framework import (
    automagic,
    configuration,
    constants,
    contexts,
    exceptions,
    interfaces,
    plugins,
)
from volatility3.framework.automagic import stacker
from volatility3.framework.configuration import requirements

# .....

framework.require_interface_version(2,0,0)
ctx = contexts.Context()

failures = framework.import_files(plugins, True)
        if failures:
            logger.warning(f"Volatility 3: The following plugins could not be loaded {''.join(sorted(failures))}")

available_automagics = automagic.available(self.ctx)
plugin_list = framework.list_plugins()

for _name, _class in plugin_list.items():
    print('Plugin name: ', _name)
    for req in _class.get_requirements():
        print('\tReq: ', req )

However, I am getting the following error :

`2023-08-03 12:37:59,066 volatility_wrapper.py WARNING Volatility 3: The following plugins could not be loaded volatility3.framework.plugins.windows.cachedumpvolatility3.framework.plugins.windows.mftscanvolatility3.framework.plugins.windows.svcscanvolatility3.framework.plugins.windows.vadyarascanvolatility3.framework.plugins.yarascan
Plugin name:  volatility3.framework.plugins.isfinfo.IsfInfo
    Req:  <ListRequirement: filter>
    Req:  <URIRequirement: isf>
    Req:  <BooleanRequirement: validate>
    Req:  <VersionRequirement: SQLiteCache>
    Req:  <BooleanRequirement: live>
Plugin name:  volatility3.framework.plugins.layerwriter.LayerWriter
    Req:  <TranslationLayerRequirement: primary>
    Req:  <IntRequirement: block_size>
    Req:  <BooleanRequirement: list>
    Req:  <ListRequirement: layers>
Plugin name:  volatility3.framework.plugins.frameworkinfo.FrameworkInfo
Plugin name:  volatility3.framework.plugins.configwriter.ConfigWriter
    Req:  <TranslationLayerRequirement: primary>
    Req:  <BooleanRequirement: extra>
Plugin name:  volatility3.framework.plugins.banners.Banners
    Req:  <TranslationLayerRequirement: primary>
Plugin name:  volatility3.framework.plugins.timeliner.Timeliner
    Req:  <BooleanRequirement: record-config>
    Req:  <ListRequirement: plugin-filter>
    Req:  <BooleanRequirement: create-bodyfile>
Plugin name:  volatility3.framework.plugins.mac.lsmod.Lsmod
    Req:  <ModuleRequirement: kernel>
Plugin name:  mac.pslist.PsList
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <ChoiceRequirement: pslist_method>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.netstat.Netstat
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <VersionRequirement: macutils>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.kevents.Kevents
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <VersionRequirement: macutils>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.proc_maps.Maps
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  mac.mount.Mount
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
Plugin name:  volatility3.framework.plugins.mac.list_files.List_Files
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: mount>
Plugin name:  volatility3.framework.plugins.mac.lsof.Lsof
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.pslist.PsList
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <ChoiceRequirement: pslist_method>
    Req:  <ListRequirement: pid>
Plugin name:  mac.lsmod.Lsmod
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.mac.check_sysctl.Check_sysctl
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.mac.vfsevents.VFSevents
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.mac.malfind.Malfind
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.ifconfig.Ifconfig
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
Plugin name:  volatility3.framework.plugins.mac.check_trap_table.Check_trap_table
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: lsmod>
    Req:  <VersionRequirement: macutils>
Plugin name:  volatility3.framework.plugins.mac.kauth_scopes.Kauth_scopes
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.mac.timers.Timers
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.mac.check_syscall.Check_syscall
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  timeliner.Timeliner
    Req:  <BooleanRequirement: record-config>
    Req:  <ListRequirement: plugin-filter>
    Req:  <BooleanRequirement: create-bodyfile>
Plugin name:  volatility3.framework.plugins.mac.bash.Bash
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  mac.kauth_scopes.Kauth_scopes
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.mac.kauth_listeners.Kauth_listeners
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
    Req:  <PluginRequirement: kauth_scopes>
Plugin name:  volatility3.framework.plugins.mac.mount.Mount
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
Plugin name:  volatility3.framework.plugins.mac.pstree.PsTree
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
Plugin name:  volatility3.framework.plugins.mac.trustedbsd.Trustedbsd
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.mac.psaux.Psaux
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.mac.socket_filters.Socket_filters
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: macutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.linux.lsmod.Lsmod
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.linux.kmsg.Kmsg
    Req:  <ModuleRequirement: kernel>
Plugin name:  linux.pslist.PsList
    Req:  <ModuleRequirement: kernel>
    Req:  <ListRequirement: pid>
    Req:  <BooleanRequirement: threads>
    Req:  <BooleanRequirement: decorate_comm>
Plugin name:  volatility3.framework.plugins.linux.pstree.PsTree
    Req:  <ModuleRequirement: kernel>
    Req:  <ListRequirement: pid>
    Req:  <BooleanRequirement: threads>
    Req:  <BooleanRequirement: decorate_comm>
Plugin name:  volatility3.framework.plugins.linux.lsof.Lsof
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <VersionRequirement: linuxutils>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.pslist.PsList
    Req:  <ModuleRequirement: kernel>
    Req:  <ListRequirement: pid>
    Req:  <BooleanRequirement: threads>
    Req:  <BooleanRequirement: decorate_comm>
Plugin name:  linux.lsmod.Lsmod
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.linux.tty_check.tty_check
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: lsmod>
    Req:  <VersionRequirement: linuxutils>
Plugin name:  volatility3.framework.plugins.linux.envars.Envars
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.malfind.Malfind
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.check_idt.Check_idt
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: linuxutils>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.linux.check_creds.Check_creds
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
Plugin name:  volatility3.framework.plugins.linux.proc.Maps
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.check_syscall.Check_syscall
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.linux.bash.Bash
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  linux.lsof.Lsof
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <VersionRequirement: linuxutils>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.sockstat.Sockstat
    Req:  <ModuleRequirement: kernel>
    Req:  <VersionRequirement: SockHandlers>
    Req:  <PluginRequirement: lsof>
    Req:  <VersionRequirement: linuxutils>
    Req:  <BooleanRequirement: unix>
    Req:  <ListRequirement: pids>
    Req:  <IntRequirement: netns>
Plugin name:  volatility3.framework.plugins.linux.check_afinfo.Check_afinfo
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.linux.check_modules.Check_modules
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: lsmod>
Plugin name:  volatility3.framework.plugins.linux.mountinfo.MountInfo
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pids>
    Req:  <ListRequirement: mntns>
    Req:  <BooleanRequirement: mount-format>
Plugin name:  volatility3.framework.plugins.linux.elfs.Elfs
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.psscan.PsScan
    Req:  <ModuleRequirement: kernel>
Plugin name:  volatility3.framework.plugins.linux.keyboard_notifiers.Keyboard_notifiers
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: lsmod>
    Req:  <VersionRequirement: linuxutils>
Plugin name:  volatility3.framework.plugins.linux.psaux.PsAux
    Req:  <ModuleRequirement: kernel>
    Req:  <PluginRequirement: pslist>
    Req:  <ListRequirement: pid>
Plugin name:  volatility3.framework.plugins.linux.iomem.IOMem
    Req:  <ModuleRequirement: kernel>
Plugin name:  windows.handles.Handles
Exception in thread Thread-15 (__sync_handler):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "<base_path</venv/lib/python3.10/site-packages/flet_core/event_handler.py", line 28, in __sync_handler
    h(r)
  File "<base_path>/panels/memory_analysis/memory_analysis_panel.py", line 153, in pick_file_result
    self.__analysis()
  File "<base_path>/panels/memory_analysis/memory_analysis_panel.py", line 181, in __analysis
    module_instance.run()
  File "<base_path>/utils/plugin.py", line 32, in run
    artifacts, data = self._decrypt(source)
  File "<base_path>/modules/bitlocker.py", line 18, in _decrypt
    vol = VolatilityWrapper()
  File "<base_path>/utils/volatility_wrapper.py", line 46, in __init__
    for req in _class.get_requirements():
  File "<base_path>/venv/lib/python3.10/site-packages/volatility3/framework/plugins/windows/handles.py", line 53, in get_requirements
    name="pslist", plugin=pslist.PsList, version=(2, 0, 0)
AttributeError: module 'volatility3.plugins.windows.pslist' has no attribute 'PsList'. Did you mean: 'List'?

As you can see it has trouble finding the PsList attribute for some reason, even though other plugins are being loaded correctly.

Additional information Volatility is installed through pip and is version 2.4.1

The docs mention that we should call the classmethod requirements for each plugin. I assume it meant to say get_requirements as there is no requirements classmethod.

I've also looked at the Vollshel and volumetric examples that are mentioned in Issue #854 but to no avail.

[ikelos]

So that is extremely strange behaviour. it sounds like you're doing all the right things? You'd probably need to set a breakpoint (import pdb; pdb.set_trace()) and then see what's going wrong when that happens (ie, what is pslist at that point?). We've explicitly made the namespaces distinct, and ensure that when importing objects, they're always imported as module.class rather than just class to avoid confusion and ensure other plugins import directly. I'm afraid I'm not sure how else to figure out what's going on?

Thank you for spotting the mistake in the documentation, that should now be fixed as of 5d4c70d5 (which should show up under latest in the readthedocs soon).

Please report back if you figure out what's going on here, because it could be helpful to others? Feel free to ask more questions here or on our slack, if you have any...

[Dionkal]

Hey thanks for your reply.

I followed your suggestion and took a deeper look into this.

I noticed that the handles plugin is the first windows plugin that is being loaded. I placed a breakpoint in the start of the get_requirements() function. I got the following results:

(Pdb) p type(pslist)
<class 'module'>
(Pdb) p type(pslist.PsList)
*** AttributeError: module 'volatility3.plugins.windows.pslist' has no attribute 'PsList'
(Pdb) p pslist
<module 'volatility3.plugins.windows.pslist' from '/..../venv/lib/python3.10/site-packages/volatility3/framework/plugins/windows/pslist.py'>

Then what I did was to modify the PluginRequirement invocation of the handles.py by removing the .PsList attribute:

requirements.PluginRequirement(
          name="pslist", plugin=pslist, version=(2, 0, 0)
)

And it worked. I then searched for other windows plugins that require the pslist plugin as well and they loaded without any problems. I put a breakpoint on the cmdline plugin in the same section and here are the results:

(Pdb) p type(pslist)
<class 'module'>
(Pdb) p type(pslist.PsList)
<class 'abc.ABCMeta'>
(Pdb) p pslist
<module 'volatility3.plugins.windows.pslist' from '/..../venv/lib/python3.10/site-packages/volatility3/framework/plugins/windows/pslist.py'>
(Pdb) p pslist.PsList
<class 'volatility3.plugins.windows.pslist.PsList'>

So every other plugin that imports pslist has access to the PsList class but not the handles plugin. I can't understand why this happens. Maybe it's because the handles plugin is the first windows plugin that is being loaded?

[ikelos]

Hmmmm, that's really strange. It's very odd it lists them as both loading from the same file. The best I can think is that's a circular import of some kind (so pslist somehow ends up loading handles which tries to load pslist and because one's in the middle of loading, it's in the module list but doesn't have the classes that are defined later)? Any chance in that breakpoint in handles you could list the contents of the pslist module (so dir(pslist)?

It would be interesting to print out what sys.modules looks like, so we can see what's been loaded already? Also, do you have any custom modules or anything in the volatility paths that might get loaded and change the import order? I can't figure out why I can recreate it on my end...

Thanks very much for helping us debug this, it's much appreciated! 5:)

[Dionkal]

Here is the dir(pslist) output in the handles.py module:

['Callable', 'Iterable', 'List', 'Type', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'constants', 'datetime', 'exceptions', 'format_hints', 'interfaces', 'intermed', 'layers', 'logging', 'renderers', 'requirements', 'utility']

and here is in the cmdline.py module:

['Callable', 'Iterable', 'List', 'PsList', 'Type', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'constants', 'datetime', 'exceptions', 'format_hints', 'interfaces', 'intermed', 'layers', 'logging', 'pe', 'renderers', 'requirements', 'timeliner', 'utility', 'vollog']

There are some attributes in the cmdline version of pslist that are not part in the handles, among them the PsList class.

I didn't change the config path of the plugins, but I did add an extra plugin directly in the framework/plugins/windows directory. I removed it and I still have the same issue.

Also from the sys.modules I can see that the correct pslist module has been loaded:

(Pdb) p sys.modules['volatility3.framework.plugins.windows.pslist']
<module 'volatility3.framework.plugins.windows.pslist' from '/........../venv/lib/python3.10/site-packages/volatility3/framework/plugins/windows/pslist.py'>

[ikelos]

Hmmmm, so going through the import list for windows.pslist:

...
from volatility3.framework.symbols import intermed
from volatility3.framework.symbols.windows.extensions import pe
...

We can see that it loads intermed ok, but then appears to be in the process of loading pe when the exception occurs, so my guess is that we might already be loading pe or something?

That only has the following imports:

import logging
from typing import Generator, Tuple

from volatility3.framework import constants, interfaces, objects
from volatility3.framework.renderers import conversion

All of which should be pretty straight forward, and it looks like constants and interfaces have already been loaded, so I guess it's during loading of objects? 5:S Objects is pretty core and doesn't do any strange importing of anything, and none of that explains why it only happen to go wrong for handles... 5:S

I guess it somewhat depends on how you're importing volatility in the first place? windows.envars appers also import framework.objects could you please check if your framework can run that one ok? You could also try ensuring that your framework imports volatility3.framework.objects early on before loading plugins, and then we can investigate why that's necessary and/or update the documentation until it's not necessary...

[Dionkal]

It seems that there is an issue on my end. I set up the project in another computer and it worked correctly. I tried to do the same on the pc that was having the issue (purged all pip caches and removed and reinstall my repository) and it still has the same issue.

The only difference I see is that the pc that runs without issue has python 3.11 installed whereas the problematic one has python 3.10.

Sorry for wasting your time.

[ikelos]

Not a waste at all. I'm surprised we use anything that fails on 3.10 though? We should support everything back to 3.7? Volatility does store its own cache (most likely ~/.cache/volatility3) you could see if that causes the problem. You might also get rid of all *.pyc files in case something got messed up on first run by python?

[Dionkal]

I downgraded python to 3.10 and still worked in the other computer. I also removed the volatility cache along with all the other related pip caches and .pyc files and downloaded with pip again. Still the same issue.

I'v also noticed something strange with how the modules are being imported.

From the list_plugins() method I get some linux modules as :

volatility3.framework.plugins.linux.proc.Maps

and some others as :

volatility3.plugins.linux.pslist.PsList

both load without any problems.

But when I get to the windows plugins the first plugins that gets loaded is the handles.py:

volatility3.plugins.windows.handles.Handles

From what I've read in the volatility __init__.py comments all plugins should be imported as volatility3. plugins othewrise they might be imported twice. But I don't import anything directly in my code I just get the plugin classes from the list_plugins method.

Update:

After a little more digging I managed to locate the issue. It indeed was my mistake. I used the volatility3.framework.plugins module in the import_files() instead of volatility3.plugins.

I followed your suggestion in #854 and took a look at the vollshell code, they seem to import both volatility3.plugins and volatility3.framework.plugins, wouldn't this cause an issue? Although they do use the volatility3.plugins in the import_files.

May I suggest an addition to the docs as well?

In the section "Determine what plugins are available", although you do provide an example on how to use the import_files() method with the volatility3.plugins, I believe that the comments in the __init__.py file about not using the volatility3.framework.plugins would be useful as well.

[ikelos]

No problem, thanks for letting us know you discovered what the issue was. I've updated the documentation to include a note to only use volatility3.plugin. Hopefully that's enough, but if you feel it needs more explanation, please let us know! 5:)