Support for adding commands via plugins in single command mode

Due to the way Typer (and by extension Click) parses arguments, we cannot pass in the name of a command that doesn't exist (yet), run a callback, then accept said command. Typer has not been designed to be extensible this way.

There are a few hacky ways to support new commands via plugins in single-command mode but none of them are great. They either require breaking some of our existing interfaces, or monkeypatching Typer/Click with some rather complex and brittle code. Below are some of the ways I've explored that could be used to support this functionality:

Load config before calling `app()`

By loading the config file before instantiating the app, we can load any plugins we find in the config, then instantiate the app. This breaks the existing --config-file option found in the callback, and will require us to manually parse this option before starting the Typer application if we want to continue providing support for custom config file locations.

def main() -> int:
    """Main entry point for the CLI."""
    # Configure logging with default settings
    # We override this later with a custom config in main_callback()
    # but we need the basic configuration in order to log to a file without leaking
    # logs to stderr for all code that runs _before_ we call configure_logging()
    # in main_callback()
    configure_logging()

+  # HACK: In order to support plugins, we need to try to load the config
+  # _before_ we call app() to ensure that the plugins are loaded.
+  # in order to load the plugins. If we don't find a config, that is ok;
+  # we will set up the config later in main_callback().
+  state = get_state()
+  config = get_config()
+  state.configure(config)
+  state._config_loaded = False  # HACK
+  load_plugins(state.config)

    try:
        app()

This approach has been confirmed to work and is currently the best candidate for implementing this, however it is also revoltingly ugly, and will require some careful consideration to implement it. Particularly the state._config_loaded = False line is such a red flag that it almost disqualifies it outright.

This also bypasses all the work that has gone into automatically creating/configuring the config file on startup, and we need to move that functionality into main() to ensure the application works as expected for new installations that do not have a config file yet.

Monkeypatch Typer and Click to hijack command parsing

It is possible to override the way the Click command group is created by Typer, and then dynamically add commands to that group. It requires some rather egregious hacks that I have not yet fully explored. At a minimum, it will require overriding typer.Typer.__call__():

class StatefulApp(typer.Typer):
    # PATCH: Override __call__ to store the click command, so we can add new commands
    # to it from plugins, etc.
    def __call__(self, *args: Any, **kwargs: Any) -> Any:
        import sys

        from typer.main import (
            _typer_developer_exception_attr_name,  # pyright: ignore[reportPrivateUsage]
        )
        from typer.main import except_hook
        from typer.models import DeveloperExceptionConfig

        if sys.excepthook != except_hook:
            sys.excepthook = except_hook
        try:
            self.click_command = self.as_click_group()
            return self.click_command(*args, **kwargs)
        except Exception as e:
            # Set a custom attribute to tell the hook to show nice exceptions for user
            # code. An alternative/first implementation was a custom exception with
            # raise custom_exc from e
            # but that means the last error shown is the custom exception, not the
            # actual error. This trick improves developer experience by showing the
            # actual error last.
            setattr(
                e,
                _typer_developer_exception_attr_name,
                DeveloperExceptionConfig(
                    pretty_exceptions_enable=self.pretty_exceptions_enable,
                    pretty_exceptions_show_locals=self.pretty_exceptions_show_locals,
                    pretty_exceptions_short=self.pretty_exceptions_short,
                ),
            )
            raise e

We then need to reference this StatefulApp.click_command when loading plugins, and ensure we use Typer's machinery for constructing commands to add the new commands to the command group. This seems brittle, but could work.

Dynamically dispatch commands via callback

It should be possible to add a catch-all arg parameter to the main callback, load plugins, then resolve whether or not arg is a valid command or not, thereby bypassing the normal Typer/Click command parsing entirely.

@app.callback(invoke_without_command=True)
def main_callback(
    ctx: typer.Context,
+  arg: Optional[str] = typer.Argument(
        None,
        help="Zabbix-CLI command to execute when running in command-line mode.",
        hidden=True,
    ),

One big problem with this approach is that it bypasses Typer's magic for providing default arguments. Furthermore, we have to deal with forwarding options from the callback to the determined function. It was messy enough to implement it for zabbix-cli --file:

https://github.com/unioslo/zabbix-cli/blob/6ff23bc987e7fbbe04d148f52e6913a404a642a0/zabbix_cli/bulk.py#L115-L149

unioslo / zabbix-cli