Isolated plugin research

[mssalvatore]

Spike

Objective

Infection Monkey needs to support plugins that have conflicting dependencies. Determine if there's an existing plugin framework or other solution that makes this possible, or develop a plan and prototype for achieving this.

Scope

• [x] Is there a third-party solution for exactly this problem? (0d) @cakekoa
• [x] Is there a third-party solution that could be repurposed to solve this problem? (0d) @cakekoa
• [x] Prototype a solution (0d)

[cakekoa]

A number of plugin frameworks exist, but none that address conflicting dependencies from plugins



We could probably use any framework as long as the plugins handle loading of the dependencies themselves, e.g., by “vendoring” dependencies



Other option: Use setup tools to install the plugin and complain about dependency problems 

 



Either way, the dependencies are pushed to the plugins, but the first option would allow otherwise conflicting dependencies to exist in different plugins

[cakekoa]

A project called python-localimport provides a context manager that allows importing from a directory

PEP582 proposes the use of a __pypackages__ directory which is automatically activated for .py files, but would only work for the script that is executed. The pythonloc and piploc executables support this PEP

A comment on a QGIS issue outlines one approach to venturing dependencies

[cakekoa]

Due to the way that python's importer imports modules, it appears that the following three methods are viable solutions:

• Run plugins in separate processes (via multiprocess)
  • A working prototype already exists
• Run plugins in separate processes (separate executables)
• Replace the builtin importer, and "namespace" the imported modules
  • This should be viable without using multiprocess
  • A custom Finder can be implemented to transform, e.g., import a to import plugin.a, and store the imported modules in sys.modules with the qualified name
  • The builtin importer will try to pull from sys.modules before running Finders. Therefore, a custom importer would need to be implemented to allow the finders to transform the names prior to default import processing

A little bit about importing:

• The call to import does the following:
  • As an optimization, it first checks to see if the module name is already in sys.modules. If so, it simply returns that module
  • A dotted-name import (e.g., import a.b.c) is split into its components. It ensures that parent packages have been imported already. So, it will first import a, then b, then c
  • The API allows for customization for these individual imports, via Finders and Loaders
  • When it loads a module, it will load that module's imports as well

[cakekoa]

We have a promising prototype in progress, but it needs the following improvements:

• The custom Finders are currently run in sequence, and have no knowledge of which dependency (and more importantly, which plugin) is doing the import. This manifests in the following way:
  • If the plugin wants to import a.b it needs to be converted to the plugin namespace. So the Finder for plugin1 will convert it to plugin1.a.b. However, if plugin2 also has an a.b, then it may have the opportunity to load plugin2.a.b instead. Since the importer stops after one success, it will have mistakenly loaded the dependency for the other plugin.
  • We should update the importer to load the dependency for the corresponding plugin only (perhaps by accessing the __file__ field in the globals parameter