gooseBit plugin/module system design

[jameshilliard]

There's a number of use cases where users of gooseBit may need to extend the updater application with custom functionality that for various reasons should be maintained separately from the upstream gooseBit project, ideally we want it to be possible to extend gooseBit without having to fork and modify the base project codebase.

For example we want to be able to implement a configuration management extension for provisioning of device keys as a gooseBit plugin/module.

In terms of how plugin configuration would look I'm thinking something like this via the yaml settings file may make sense:

plugins:
  upstreamdata.cfd.provisioner:
    configkey1: configval1
    configkey2: configval2
    configkey3:
      subkey1: subval1
      subkey2: subval2

The idea would be that we have a python class/package with the import name upstream.cfd.provisioner that extends some sort of base module class from gooseBit which would be passed the decoded plugin child yaml configuration structure(a multilevel dictionary in the above example) when the module __init__ method is called during plugin load after dynamically importing the upstream.cfd.provisioner python module class.

The module class implementation from the plugin project would then be expected to provide implementations for the various hook methods/variables that gooseBit would call in various places such as when the device sends config vars or when device poll various endpoints.

The module system would also likely need a way for plugins to define their own database models/tables which may need to reference the existing gooseBit models so would likely need orm/migration hooks for integrating that as well. Likely we want these plugin defined database models/tables to have some sort of namespacing pattern that ensures they won't conflict with other plugins or the base gooseBit project models/tables.

Some potentially relevant info: https://github.com/team23/fastapi-module-loader https://github.com/Renumics/spotlight/blob/main/renumics/spotlight/plugin_loader.py https://fastapi.tiangolo.com/tutorial/dependencies/#fastapi-plug-ins https://fastapi.tiangolo.com/features/#unlimited-plug-ins https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/

[b-rowan]

The module system would also likely need a way for plugins to define their own database models/tables which may need to reference the existing gooseBit models so would likely need orm/migration hooks for integrating that as well. Likely we want these plugin defined database models/tables to have some sort of namespacing pattern that ensures they won't conflict with other plugins or the base gooseBit project models/tables.

This can be accomplished by loading plugins before initializing the DB, and appending all plugin modules to the tortoise configuration, here - https://github.com/UpstreamDataInc/goosebit/blob/714ad04b9090fe1585a95cbc43f055c04591aebd/goosebit/db.py#L6-L13

[b-rowan]

I imagine you could have a configuration that looks something like this:

plugins:
  goosebit_plugins.cfd.provisioner:
    db:
      models: goosebit_plugins.cfd.provisioner.db.models
    ...

[jameshilliard]

I imagine you could have a configuration that looks something like this:

I'm thinking the models should be defined by a plugin class property or something, not something that's passed by the config file.

[jameshilliard]

This can be accomplished by loading plugins before initializing the DB, and appending all plugin modules to the tortoise configuration, here -

Yeah, I think maybe just iterate all plugin classes and check each for a models property or something and append from that.

[tsagadar]

I never implemented a plug-in mechanism. But besides the technical challenges mentioned above I think it also needs careful modeling of the extension points. This could be tricky to get right upfront. Maybe better to regularly implement the feature and then see if it could be extracted as a plug-in?

[jameshilliard]

This could be tricky to get right upfront.

Yeah, figured I would make open an issue to figure out an initial design. At least initially having a stable plugin API probably isn't super important so we can hopefully iterate on the API design a bit before stabilizing it.

Maybe better to regularly implement the feature and then see if it could be extracted as a plug-in?

Technically we have already implemented the cfd provisioner system as part of the original internal project gooseBit is based off of. At a minimum I think a plugin like the cfd provisioner will need a way to add its own database models/tables that reference base gooseBit tables and a way to add additional pages to the webui. It will also need a way to add request handling hooks to the various DDI API endpoints that can conditionally override default logic such as for overriding the update file being served and receiving config vars.

[b-rowan]

I have created a very simple PR for now that should demonstrate what I believe is the best way to do this, following https://setuptools.pypa.io/en/latest/userguide/entry_point.html

I for some reason cannot get this to work, and have tried moving around directories to not be stacked for the example plugin, and also renaming the goosebit.plugins module to make sure the naming doesn't overlap. Not sure if anyone else has ideas on this side of things?

[jameshilliard]

I have created a very simple PR for now that should demonstrate what I believe is the best way to do this, following https://setuptools.pypa.io/en/latest/userguide/entry_point.html

Hmm, I'm not sure this is the best approach, at least not by itself, it's unclear how we would say have a plugin that exposes its own configuration options and such here in a way that would tie in nicely.