juju-solutions / matrix

Automatic testing of big software deployments under various failure conditions
Other
8 stars 9 forks source link

Support tests and tasks from bundle under test #26

Closed johnsca closed 7 years ago

johnsca commented 7 years ago

Currently unable to test due to juju-solutions/python-libjuju#7 and https://bugs.launchpad.net/juju/+bug/1645917

johnsca commented 7 years ago

Rendered --help:

usage: matrix [-h] [-l LOG_LEVEL] [-L [LOG_NAME [LOG_NAME ...]]]
              [-f [LOG_FILTER [LOG_FILTER ...]]] [-s {tui,raw}] [-x FILENAME]
              [-i INTERVAL] [-p PATH] [-D] [-B]
              [-t [TEST_PATTERN [TEST_PATTERN ...]]] [-g GLITCH_PLAN]
              [-n GLITCH_NUM] [-o GLITCH_OUTPUT]
              [additional_suites [additional_suites ...]]

Run a local bundle through a suite of tests to ensure that it can handle the
types of operations and failures that are common for all deployments.

positional arguments:
  additional_suites     Additional suites to be merged with the default suite
                        before running

optional arguments:
  -h, --help            show this help message and exit
  -l LOG_LEVEL, --log-level LOG_LEVEL
  -L [LOG_NAME [LOG_NAME ...]], --log-name [LOG_NAME [LOG_NAME ...]]
  -f [LOG_FILTER [LOG_FILTER ...]], --log-filter [LOG_FILTER [LOG_FILTER ...]]
  -s {tui,raw}, --skin {tui,raw}
  -x FILENAME, --xunit FILENAME
                        Create an XUnit report file
  -i INTERVAL, --interval INTERVAL
  -p PATH, --path PATH  Path to local bundle to test (defaults to current
                        directory)
  -D, --no-default-suite
                        Do not include the default suite as the first suite
  -B, --no-bundle-suite
                        Do not include the suite provided by the bundle
  -t [TEST_PATTERN [TEST_PATTERN ...]], --test_pattern [TEST_PATTERN [TEST_PATTERN ...]]
                        Pattern for selecting which tests from the suite(s)
                        are run
  -g GLITCH_PLAN, --glitch_plan GLITCH_PLAN
  -n GLITCH_NUM, --glitch_num GLITCH_NUM
  -o GLITCH_OUTPUT, --glitch_output GLITCH_OUTPUT

A default suite of tests will always be run, unless -D is given.  Additional
suites can be passed in as arguments.  Additionally, the bundle will be
checked for a tests/matrix.yaml file which will be included as an additional
suite if present.  All suites will be merged together, and suites containing
tests with the same name as a test in a previous suite will override that
test.

Examples:

    Run all normal tests (default suite and, if present, ./tests/matrix.yaml)
    on the bundle at ./:

        $ matrix

    Run only the default suite (ignore ./tests/matrix.yaml):

        $ matrix -B

    Run only ~/foo/tests/matrix.yaml on the bundle at ~/foo:

        $ matrix -Dp ~/foo

    Run all normal tests plus ./tests/matrix_extra.yaml on the bundle at ./:

        $ matrix tests/matrix_extra.yaml

    Run only ./tests/matrix_extra.yaml:

        $ matrix -DB tests/matrix_extra.yaml
pengale commented 7 years ago

LGTM/+1