cli: external tool discoverability

[richardjgowers]

Define how the auto-documentation of yaml files will behave.

Should define

• how help is requested by user
• what help is provided by programs
• how tools can define their available help messages

[richardjgowers]

tool discoverability seems pretty easy: https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata

e.g. if I put this line in kartograf's pyproject.toml:

[project.entry-points."openfe.mappers"]
Kartman = "kartograf.atom_mapper.KartografAtomMapper"

then in python I can do:

In [6]: metadata.entry_points(group='openfe.mappers')
Out[6]: [EntryPoint(name='Kartman', value='kartograf.atom_mapper.KartografAtomMapper', group='openfe.mappers')]

and life is good.

[dwhswenson]

The approach that is already built into plugcli is the other one on that page: https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-namespace-packages The plugcli plugins are also inspired by this: https://click.palletsprojects.com/en/8.1.x/commands/#custom-multi-commands

The advantage of this approach is that you can also easily allow "quick and dirty" plugins. A simple, self-contained plugin can be loaded from a file without needing to go through packaging. This is great for the kinds of users who are uncomfortable with Python packaging (or as a first step before properly packaging something up).

On the user end, this is just a matter of

# in a module in the openfe_plugins namespace, or in a file to be placed in the appropriate location
from openfecli.plugins import AtomMapperPlugin
# (or maybe this is YAMLPlugin, and requires a parameter to specify that it is an atom mapper)

PLUGIN = AtomMapperPlugin(...)

with some parameters in ... that link to the actual functionality.

On our end, you can collect all plugins with:

from plugcli.plugin_management import NamespacePluginLoader

ns_loader = NamespacePluginLoader("openfe_plugins", AtomMapperPlugin)
plugins = ns_loader()

To also support quick and dirty plugins in ~/.openfe/plugins/ (although we shouldn't hard-code that directory; use tools that give the correct system-specific directories):

from plugcli.plugin_management import FilePluginLoader

file_loader = FilePluginLoader("~/.openfe/plugins", AtomMapperPlugin)

plugins.append(file_loader())

[richardjgowers]

Wouldn't the user package also have to do some packaging work to get their Plugin to appear under the openfe_plugins namespace?

[dwhswenson]

Two mechanisms for plugin registration:

1. Quick and dirty. File based. No packaging needed. Put file in something like ~/.openfe/plugins/my_plugin.py
2. Package based -- if a plugin in stable enough that versions matter, this should be used. Packaging involves:
   • use find_namespace: for options.packages
   • put the plugin in openfe_plugins/my_plugin/__init__.py

[dwhswenson]

An example where we did this: https://github.com/dwhswenson/fabulous-paths/

Two comments on that

1. It is based on an old infrastructure that required a module to define variables CLI, SECTION, and OPS_VERSION: you'll see those imported in paths_cli_plugins/fabulous/__init__.py. The new approach would just put the plugin instance in __init__.py.
2. The science side of that never got fully hooked up, although the infrastructure was working fine.