Extend plugin mechanism to allow dynamic creation of check suites

[joesingo]

For the YAML work it was suggested the YAML-specific stuff live in a plugin separate to compliance-checker, and that the plugin be 'called' in some way to generate the checks when needed.

The plugin will need to access command-line arguments somehow (e.g. to get the YAML filenames)

[joesingo]

See the generator-plugins branch:

commit 85c3708a67c565d71e0b95503fb46421be320add
Author: Joe Singleton <joesingo@gmail.com>
Date:   Mon Mar 26 14:23:01 2018 +0000

    Add support for a new type of plugin that can dynamically generate check suites

    - Through 'compliance_checker.generator' entry point, plugins can point
      to a class with methods

      - add_arguments(parser): take an argparse parser object and add any
        necessary command line arguments

      - get_suites(args): take command line arguments and return a dict
        mapping suite name to checker class

A minimal example is as follows:

setup.py:

from setuptools import setup, find_packages

setup(
    name         = "example-plugin",
    description  = "Example for a plugin that dynamically generates check suites",
    packages     = find_packages(),
    entry_points = {
        "compliance_checker.generators": [
            "example-plugin = example_plugin.plugin:ExamplePlugin"
        ]
    }
)

example_plugin/plugin.py:

from compliance_checker.base import BaseCheck, Dataset

class ExamplePlugin(object):
    @classmethod
    def add_arguments(self, parser):
        # Plugin can add arguments to command-line tool
        parser.add_argument("-e", "--extra-arg", help="Some extra argument")

    @classmethod
    def get_suites(self, args):
        # Here the plugin can use `args` to generate the check suite in a
        # dynamic way. In this example just return the same suite regardless
        return {"generated-checks": ExampleSuite}

class ExampleSuite(BaseCheck):
    supported_ds = [Dataset]

    def setup(self, ds):
        pass

    def check_something(self, ds):
        return True

If this is installed as a package (i.e. touch example_plugin/__init__.py && pip install .) then you can:

• Run compliance-checker -h to observe an option -e EXTRA_ARG, --extra-arg EXTRA_ARG has been added
• Run compliance-checker -t generated-checks <dataset> to observe the check has been registered and run

See cc-yaml for a 'real' example that takes a YAML file on the command line and generates a check suite from its contents.