Fix default plugins command line option

[GoogleCodeExporter]

Hey guys, 

The help for --plugins says: Additional plugin directories to use (colon 
separated)

So I expected that specifying --plugins=mydir would let me load plugins from 
the default (./plugins) in addition to mydir. But it looks like if you specify 
--plugins=mydir on command-line, it only searches mydir and not ./plugins. 

If that's how you intend it to work, then its OK but I have a feeling it should 
work differently. The default value could be changed to none:

config.add_option("PLUGINS", default = None,
                  help = "Additional plugin directories to use (colon separated)")

and then something like:

# Setup initial plugin directories
plugins = './plugins'

if config.PLUGINS != None:
      plugins += ':{0}'.format(config.PLUGINS)

for path in plugins.split(':'):
   ......

Original issue reported on code.google.com by michael.hale@gmail.com on 26 Aug 2010 at 1:57

[GoogleCodeExporter]

I believe it's by intention, to allow you to swap out whole libraries of 
plugins/change the core ones whilst still maintaining a clean original set.  
Note that configuration options can be set permanently in ~/.volatilityrc (and 
I think /etc/volatilityrc, but don't quote me on that), if you're intending to 
use your own plugins directory often.

The plugins option needs looking at due to where volatility currently installs 
itself (/usr/bin/volatility.py) allowing it to be run from any directory (which 
obviously then won't find ./plugins).  The reason we used ./plugins was to try 
and get an easy cross-platform win, but it probably needs fixing.  Currently 
the core plugins aren't made a part of the volatility.* namespace, but they 
could be if necessary?

I've changed the summary to reflect this, but where plugins are absolute or in 
addition to core plugins (and whether core plugins should be classed as 
additional data files or as part of the volatility namespace in site-packages) 
is a design decision so I've CCed some of the other to get a consensus.  Guys, 
please place your thoughts in here, and if there's any strong feelings one way 
or the other then when I come to fix this I'll make the appropriate changes...

Original comment by mike.auty@gmail.com on 26 Aug 2010 at 11:07

• Changed title: Fix default plugins command line option
• Changed state: Accepted
• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

We basically scan all the plugin directories specified in the parameter (which 
can also be defined in any configuration file as specified by the --conf-file 
parameter). If you want to scan more than one directory you can use : to 
separate them.

Its possible to specify a default value for PLUGINS based on the current module 
path - e.g.
config.add_option("PLUGINS", default=os.path.dirname(__file__)+"/../plugins",
                  help = "Additional plugin directories to use (colon separated)")

To help it find the default plugin path (but only if its not overriden in the 
command line or the config file).

To make it runnable from any directory we need to add the main volatility.py 
path to the sys.paths so modules can be found relative to it:

import sys

sys.path.append(os.path.dirname(__file__))

But that might interfere with py2exe or other packaging systems.

Of course this all goes away if we use the proper distutil installer where the 
packages will be put in the python-support and we can write a config file in 
/etc/volatilityrc.

Original comment by scude...@gmail.com on 26 Aug 2010 at 11:28

[GoogleCodeExporter]

We do have a setup.py distutils installer, and it currently puts the volatility 
framework (without plugins) into site-packages, and it puts plugins into a data 
directory (/usr/share/volatility).  That same file is used to build the py2exe 
extension, and so having the plugins as data files allows them to be left 
outside the exe (meaning people aren't stuck with code/plugins/etc with issues, 
if they want to fix them themselves).  I haven't tried using the setup.py to 
create a python installer (rather than a python binary), so I dunno how that'll 
work out yet.  People seemed to be pushing for a single file they could take 
wherever.

There's also the problem of cross-directory plugin reuse.  This showed itself 
when I moved verinfo (unsupported!) to the contrib directory.  Since it tries 
to import procdump, which is no longer in the same directory, it fails right 
from the start.  I'm not sure how to fix this other than instantiating the 
plugin repository and look up required modules in there.  Again, not something 
I want to face for the deadlines we're trying to achieve for 1.4.

So scudette, I couldn't clearly make out your opinions in that last post?  It 
sounds like you want the core plugins to always have to be specified (so they 
can be overridden), is that right?

As to the __FILE__ use, I'll look into how that affects py2exe and so on...

Original comment by mike.auty@gmail.com on 26 Aug 2010 at 12:03

[GoogleCodeExporter]

Ok, so now that we've got the module loading sorted, this should be relatively 
easy to solve.  Unfortunately, currently the files are all installed into 
/usr/share/volatility/plugins at the moment.  This should be easy to fix, by 
simply removing the separation, so all the core plugins would actually live 
under volatility.plugins, but since modules can have multiple paths, we can 
still load external plugins.  The default path would then become:

os.path.join(os.path.dirname(__FILE__), "plugins")

This would also mean that for core plugins (and contributed ones importing 
them) any static analysis will find them in the right place, so it'll all work 
out by default.

Can anyone see any downsides with this?  And reason to keep the plugins so 
distinct from the framework (rather than just within their own namespace)?

That also leaves the question of whether we should allow people to set a 
plugins directory that does *not* include the core plugins?  Something inside 
of me says we should (why force core plugins on people?), but I haven't figured 
out if it will confuse people setting their own plugin dirs?  Again 
comments/suggestions welcome...

Original comment by mike.auty@gmail.com on 9 Nov 2010 at 1:36

[GoogleCodeExporter]

I think we should allow people to set _additional_ plugin directories on top of 
the core ones (which should be always loaded). The core plugins provide some 
very important things (like critical overlays etc) which are always useful. In 
that case maybe we should force a load of the core plugins from the regular 
namespace - i.e. write an __init__.py which imports all the plugins in the 
plugin directory and explicitly do import volatility.plugins. This is not as 
dynamic but will ensure that core plugins are always loaded even when we use 
py2exe.

We can then take a plugin command line arg for loading additional directories.

Other than that I love the new plugin system.

Original comment by scude...@gmail.com on 9 Nov 2010 at 7:45

[GoogleCodeExporter]

Ok, well, it took a little while, but I've got a patch drawn up that seems to 
fulfill all our requirements.

This patch means that the core plugins will always be loaded.  Also the plugin 
loader now accepts zipfiles, so 
--plugins="/path/to/zipfile.zip/path/within/zipfile/to/plugins" would import 
all plugins (.py{,c,o}) in zipfile.zip within /path/within/zipfile/to/plugins.  
Note that just like with a normal directory, __init__ files are necessary to 
locate submodules.

The upshot is that this is exactly how py2exe works, so the __file__ entry of 
any file will read C:\path\to\volatility.exe\volatiltiy\file.py, meaning we can 
use it to find the core plugins when volatility's both a source package, and an 
exe.

The reason I haven't committed this directly is because it moves all the plugin 
files, and more importantly, since the plugin importer now looks for .pyc and 
.pyo files, any of those left lying about will be re-imported, potentially 
causing problems.  They're necessary for py2exe (since that doesn't store the 
source files), but I'm wondering if we should only search for .py files when 
going through sources?

Either way, this could do with a look over (the patch without all the file 
moves, which I've attached, is quite short)...

Original comment by mike.auty@gmail.com on 9 Nov 2010 at 10:10

Attachments:

• volatility-zip-plugins.patch

[GoogleCodeExporter]

Ok, no complaints (before it was applied), so I've pushed it in order to get on 
with other patches.  If there's any problems, I expect we'll see them soon 
enough.  The biggest one I'm expecting is it locating people's old .pyc/.pyo 
files, so anyone reading this, please clear them out first...  5:)

Original comment by mike.auty@gmail.com on 11 Nov 2010 at 10:40

• Changed state: Fixed