Mechanism to control which plugins are executed

[simonw]

DJP currently executes any plugin that that has been installed using pip install, detected via Pluggy's support for entry points.

Some users may want to explicitly select the plugins that are active.

This could be achieved using an optional PLUGINS setting, which can be a list of plugin names that the user wants to be active.

[simonw]

A related feature could be a PLUGINS_DIR=path setting, which scans that directory for plugins to load (like the Datasette --plugins-dir option). This would be really useful for developing new plugins too.

[pirate]

I implemented a solution to this already, you're welcome to borrow part/all/none of it.

settings.py:

PLUGIN_HOOKSPECS = [
    'djp.hookspec',          # DJP's default hooks like settings(), middleware(), etc.
    'myapp.auth.hookspec',
    'myapp.forms.hookspec',
    'myapp.themes.hookspec',
]
djp.register_hookspecs(PLUGIN_HOOKSPECS)
# useful e.g. if your app defines multiple hookspecs for different areas, nice to have it declaratively listed and to be able to add/remove some for testing
# for more complex apps this also lets you do plugin meta-fun where plugins can
# expose new hookspecs for other plugins (but imo they should still be listed flat here, not dynamically discovered, otherwise it becomes insanity)

# module name prefix : plugin category dir path on filesystem
# (individual plugins will be found within each plugin category dir)
PACKAGE_DIR = Path(__file__).resolve().parent.parent  # source code root dir
BUILTIN_PLUGIN_DIRS = {
    'auth_plugins':              PACKAGE_DIR / 'auth_plugins',
    'form_plugins':              PACKAGE_DIR / 'form_plugins',
    'theme_plugins':             PACKAGE_DIR / 'theme_plugins',
    # ...
}
USER_PLUGIN_DIRS = {
    # Example: add any dirs to pull plugins from here
    # 'user_plugins':            archivebox.DATA_DIR / 'user_plugins',
    'user_plugins':              Path(os.environ.get('DJP_PLUGINS_DIR'),
}

BUILTIN_PLUGINS = djp.get_plugins_in_dirs(BUILTIN_PLUGIN_DIRS)
PIP_PLUGINS = djp.get_pip_installed_plugins(group='archivebox')  # pip entrypoint group to look for
USER_PLUGINS = djp.get_plugins_in_dirs(USER_PLUGIN_DIRS)
ALL_PLUGINS = {**BUILTIN_PLUGINS, **PIP_PLUGINS, **USER_PLUGINS}

PLUGIN_MANAGER = djp.pm
PLUGINS = djp.load_plugins(ALL_PLUGINS)

djp/__init__.py:

def register_hookspecs(hookspecs):
    for hookspec_import_path in hookspecs:
        hookspec_module = importlib.import_module(hookspec_import_path)
        pm.add_hookspecs(hookspec_module)

def find_plugins_in_dir(plugins_dir: Path, prefix: str) -> Dict[str, Path]:
    return {
        f"{prefix}.{plugin_entrypoint.parent.name}": plugin_entrypoint.parent
        for plugin_entrypoint in sorted(plugins_dir.glob("*/apps.py"))  # key=get_plugin_order  # see note below 
    }   # "plugins_pkg.pip": "/app/archivebox/plugins_pkg/pip"

def get_pip_installed_plugins(group='djp'):
    """replaces pm.load_setuptools_entrypoints("djp")"""
    import importlib.metadata

    DETECTED_PLUGINS = {}   # module_name: module_dir_path
    for dist in list(importlib.metadata.distributions()):
        for entrypoint in dist.entry_points:
            if entrypoint.group != group or pm.is_blocked(entrypoint.name):
                continue
            DETECTED_PLUGINS[entrypoint.name] = Path(entrypoint.load().__file__).parent
            # pm.register(plugin, name=ep.name)
            # ^ dont do this now, we wait till load_plugins() is called
    return DETECTED_PLUGINS

def get_plugins_in_dirs(plugin_dirs: Dict[str, Path]):
    DETECTED_PLUGINS = {}
    for plugin_prefix, plugin_dir in plugin_dirs.items():
        DETECTED_PLUGINS.update(find_plugins_in_dir(plugin_dir, prefix=plugin_prefix))
    return DETECTED_PLUGINS

def load_plugins(plugins_dict: Dict[str, Path]):
    LOADED_PLUGINS = {}
    for plugin_module, plugin_dir in plugins_dict.items():
        # print(f'Loading plugin: {plugin_module} from {plugin_dir}')
        plugin_module_loaded = importlib.import_module(plugin_module + '.apps')
        pm.register(plugin_module_loaded)  # assumes hookimpls are imported in plugin/apps.py
        LOADED_PLUGINS[plugin_module] = plugin_module_loaded.PLUGIN
        # print(f'    √ Loaded plugin: {plugin_module}')
    return LOADED_PLUGINS

Generally I find having the list of loaded plugins stored in settings.PLUGINS super helpful for areas of the app to be able to list/display/query.

Note: Someday enforcing plugin import order within a dir may be required, but right now it's not needed for me personally, order is defined by sorting the dirs in the ALL_PLUGINS dict above. I think this pushes people towards sanely splitting plugins into categories and only having order needed to be defined between categories. This way plugins dont need to manage their preferred order O(n^2) with every other plugin, they can just say "I'm a type of plugin that should be loaded with this whole category abc, which comes before this whole other category xyz".

[kfdm]

This may be offtopic (in which case ignore) but checking the DEBUG setting for certain ones to enable/disable may be useful. Examples like django-debug-toolbar (though this may check DEBUG internal to the plugin)

[tjltjl]

Some users may want to explicitly select the plugins that are active

ALL users should probably want to try to avoid the potential supply chain attack surface. Installing by default extensions from any multi-level dependency seems risky. (just my first reaction on reading about djp - avoiding friction is cool but "explicit is better than implicit")

[pirate]

Yeah I agree @tjltjl, I think there should be a manual line in settings.py or somewhere to explicitly load plugins from auto-discovered packages:

PIP_PLUGINS = djp.get_pip_installed_plugins(group='archivebox')  # pip entrypoint group to look for

Not just for security reasons, also to help developers trace where code is getting loaded and run from.