Add support for sub-plugins architecture associated (registered) to a standard top-level plugin

[ganeshrn]

Proposal:

Author: Ganesh B. Nalawade <@ganeshrn>

Date: 2021-06-28

• Status: New
• Proposal type: (core design, process, ?)
• Targeted release: <future release #, ex. 2.2>
• Associated PR: <link to GH PR in ansible/proposals if PR was submitted>
• Estimated time to implement: <X days, weeks, etc.>

Motivation

Describe the reasons for this proposal.

The current Ansible core architecture works well in case the plugin has 1:1 mapping that is all the functionality is contained within the plugin and is not required to invoke other plugins but not so well when the mapping is 1:N which is applicable mainly for network use case.

Problems

What problems exist that this proposal will solve? In the current plugin architecture, a top-level plugin invokes another related top-level plugin, for example, the ansible.netcommon.network_cli connection plugin is used to connect to different vendor network OS. To support different customization for each platform the terminal and cliconf plugins were implemented as top-level plugins even though they are not directly used and are sub-plugins of the connection plugin (plugins invoked from another plugin). This design restricts adding new functionality requiring 1:N mapping for example grpc connection support

Solution proposal

Ansible collection allows python module import from the custom path within the same collection as well as another collection from a standard plugin. A predefined directory structure can be defined within the collection to support sub plugins. This design pattern is already used in ansible.utils and ansible.netcommon collections Reference sub-plugin loading in ansible.utils collection. This directory structure was approved before Ansible 3.0.0 release.

- name: Run command and parse with ntc_templates
  ansible.utils.cli_parse:
    command: "show interface"
    parser:
      name: ansible.netcommon.ntc_templates
  register: parser_output

In above example ansible.netcommon.ntc_templates is a sub-plugin of ansible.utils.cli_parse top-level plugin.

This however doesn't support setting sub-plugin configuration options from ansible-core and rendering sub-plugin docs.

Testing (optional)

Does / should this require testing, and if so, how? Describe here. This section is optional but could be helpful. Add sanity test for sub-plugin documentation

Documentation (optional)

Does / should this require documentation? If so, describe here. This section is optional but could be helpful. Add support to render and display sub-plugin documentation using ansible-doc and collection docs on https://docs.ansible.com/ansible/latest/collections/index.html

Anything else?

If you'd like to add anything else beyond the above required and optional sections, you are welcome to do so.

[felixfontein]

I like the idea of having a standardized way of having subplugins, and I think in hindsight it would have been good to use something like that for cliconf and terminal plugins.

Some thoughts (which are not yet complete enough to be called good ideas, but maybe someone wants to pick them up :) ):

1. How about indicating the 'owning' plugin by a subdirectory? Like plugins in plugins/sub_plugins/ansible.utils.cli_parse/ for sub-plugins of ansible.utils.cli_parse?
2. Should the plugin type of the owning plugin be mentioned somehow? If there are multiple different plugins called ansible.utils.cli_parse, it's not clear to me how the docs build process should associated subplugins to the correct plugin named ansible.utils.cli_parse.
3. I think subplugins should also be handled by the plugin loader, so that meta/runtime.yml redirects work for them. Right now I guess only symlink redirects work, but these cannot cross collection boundaries.

[felixfontein]

Though, when thinking about it, I think redirects already work with meta/runtime.yml when using the import_redirection feature, since the plugin loader is already working in the background since it hooks into Python's module loader.

[ganeshrn]

How about indicating the 'owning' plugin by a subdirectory? Like plugins in plugins/sub_plugins/ansible.utils.cli_parse/ for sub-plugins of ansible.utils.cli_parse?

Currently in ansible.utils collection the directory structure is plugins/sub_pugins/validate and the plugins under validate can be invoked from more than one top-level plugins example test and lookup.

Should the plugin type of the owning plugin be mentioned somehow? If there are multiple different plugins called ansible.utils.cli_parse, it's not clear to me how the docs build process should associated subplugins to the correct plugin named ansible.utils.cli_parse.

One way to do that is to add the information in sub-plugin documentation detailing which top-level plugins the sub-plugin is associated with. Something like

DOCUMENTATION = """
    author:  <name of author>
    name: <sub-plugin name> (eg. jsonschema)
    sub_plugin_of:
      name: validate
      types:
      - test
      - lookup
    short_description: Define configurable options for jsonschema validate sub-plugin