Test cases for components

[Guslington]

Thinking about creating test cases with config to run in travis-ci builds. For example the sqs component has some default config to create some demo queues, where as that should be handled with a tests case defined in sqs.tests.yaml. Then maybe a seperate command cftest or cfcompile --config sqs.tests.yaml.

[Guslington]

Example test yaml file with multiple test cases bellow. My thoughts were if we load each test case as seperate config, and compile the component into out/tests/#{test_case_name}.compiled.yaml or out/tests/#{test_case_name}/sqs.compiled.yaml. from there we can validate and run cfn_nag on each test case. Each test case should also load in the default config from the component to simulate a downstream project config.

queues_with_default_config:
  queues:
    - name: queue1
    - name: queue2
    - name: queue3

queues_with_config_overrides:
  queues:
    -
      name: queue1
      visibility_timeout: 60
      delay_seconds: 20
      maximum_message_size: 1024
      message_retention_period: 300
      receive_message_wait_time_seconds: 120
    -
      name: queue2
      visibility_timeout: 120
      delay_seconds: 60
      maximum_message_size: 256
      message_retention_period: 84600
      receive_message_wait_time_seconds: 600

queues_with_fifo:
  queues:
    -
      name: queue1
      fifo_queue: true
      content_based_deduplication: true
    -
      name: queue2
      fifo_queue: true

queues_with_dlq:
  queues:
    -
      name: my-queue
      message_retention_period: 600
      redrive_policy:
        queue: my-queue-dlq
        count: 10
    -
      name: my-queue-dlq
      message_retention_period: 84600

This could then print out a report and a total test coverage percentage.

======================
  CfHighlander Tests
======================
Total: 4
Pass: 3
Failure: 1

Errors:
    queues_with_dlq => 
        Validation => Template format error: Unresolved resource dependencies
======================
test coverage: 95%
======================

[Guslington]

@toshke @aaronwalker I've implemented the cftest command on my for fork and some test cases on the sqs component in sqs.tests.yaml

This is pretty basic testing, loading in test case config and looping over each test case generating a template and outputing to out/test/{test_case}/sqs.compiled/yaml. Then running validate and catching any validate errors.

Then print out the status at the end with the correct error code

  ============================
  #    CfHighlander Tests    #
  ============================

  Pass: 3
  Fail: 1

  ======= FAILURES ========
  Test: queues_with_dlq
  Type: Validation
  Message: Template format error: Resource name my-queue is non alphanumeric.

[toshke]

@Guslington

It is great idea overall, stepping quality gate a notch up from this component compiles on latest developer version would be good. Since there is separate command to test the components, i'm not worried about potential name collisions etc.

Few comments though,

1. Not sure if correct approach, but rather to enforce file name to componentname.tests.yaml, I would make this default behaviour with allowing any file with *.tests.yaml within component folder to be acceptable as test file. With this approach, each file would be just a config, rather than map of configs. It could be just me, but I like different tests split into separate files. Think of SCM implications of having 10 tests and single file, would be hard to track history of single test case

2. https://github.com/theonestack/cfhighlander/compare/develop...Guslington:feature/component-test?expand=1#diff-e3f178d0aee7237d7ffe28dda9c0f37dR202

Since testing will be headless most of the time, I would just enforce silent mode here.

[aaronwalker]

@toshke @Guslington My thoughts is that the test config files should be in a tests/ directory to make it easier to see the default config regardless

[Guslington]

@toshke @aaronwalker I've updated the fork with the changes to get tests from a sub directory. This will default to tests/ but can be overridden using the -d,--directory option. You can also pass in an array of tests using -t,--test using the relative path. -t tests/x.test.yaml tests/y.tests.yaml

I've update the sqs branch with example test cases.

I have also added a metadata section to add some details about the test. The test process uses the name section for reporting.

test_metadata:
  type: config
  name: my test name
  description: |
    This is a long description of
    my test.

I will add the doco to the readme once we're happy with it and probably a shortcut command for cftest

[aaronwalker]

@Guslington we might also want to add option for outputting test result in junit xml format so it can be reported on.

[Guslington]

@aaronwalker now generating xml and json reports with the -r --report [json,xml] option

XML

<?xml version="1.0" encoding="UTF-8"?>
<testsuite component="sqs" tests="4" pass="4" failures="0" time="13.618055" timestamp="2019-02-05T16:42:54.825+1100" >
    <testcase name="fifo queues" test="tests/fifo.test.yaml" type="compile"  time="0.012548"/>
    <testcase name="fifo queues" test="tests/fifo.test.yaml" type="Validation"  time="12.729442"/>
    <testcase name="dlq" test="tests/dlq.test.yaml" type="compile"  time="0.009314"/>
    <testcase name="dlq" test="tests/dlq.test.yaml" type="Validation"  time="0.635338"/>
    <testcase name="queues with default config" test="tests/default_config.test.yaml" type="compile"  time="0.009369"/>
    <testcase name="queues with default config" test="tests/default_config.test.yaml" type="Validation"  time="0.199935"/>
    <testcase name="queues with config overrides" test="tests/config_overrides.test.yaml" type="compile"  time="0.006334">
        <failure message="undefined method `MaximumMessageSizde' for #<CfnDsl::AWS::Types::AWS_SQS_Queue:0x00007faee48b1ec0>" type="Cfhighlander::Compiler::ComponentCompiler"/>
    </testcase>
</testsuite>

JSON

{
  "component": "sqs",
  "tests": "4",
  "pass": "4",
  "failures": "0",
  "time": "0.387949",
  "timestamp": "2019-02-05T16:39:01.475+1100",
  "testcases": [
    {
      "name": "fifo queues",
      "test": "tests/fifo.test.yaml",
      "type": "compile",
      "failure": false,
      "time": "0.006653"
    },
    {
      "name": "fifo queues",
      "test": "tests/fifo.test.yaml",
      "type": "Validation",
      "failure": false,
      "time": "0.125685"
    },
    {
      "name": "dlq",
      "test": "tests/dlq.test.yaml",
      "type": "compile",
      "failure": false,
      "time": "0.008255"
    },
    {
      "name": "dlq",
      "test": "tests/dlq.test.yaml",
      "type": "Validation",
      "failure": false,
      "time": "0.068771"
    },
    {
      "name": "queues with default config",
      "test": "tests/default_config.test.yaml",
      "type": "compile",
      "failure": false,
      "time": "0.008193"
    },
    {
      "name": "queues with default config",
      "test": "tests/default_config.test.yaml",
      "type": "Validation",
      "failure": false,
      "time": "0.07563"
    },
    {
      "name": "queues with config overrides",
      "test": "tests/config_overrides.test.yaml",
      "type": "compile",
      "failure": false,
      "time": "0.008565"
    },
    {
      "name": "queues with config overrides",
      "test": "tests/config_overrides.test.yaml",
      "type": "Validation",
      "failure": false,
      "time": "0.077245"
    }
  ]
}