Integrate run unit and run e2e into run terraform

[eistrati]

It should be as simple as configuring .recink.yml, similar to below example:

tf_script:
  root: './tf_script'
  terraform:
    version: 0.11.0
    plan: true
    apply: true
    destroy: true
    test:
      mapping: './tests/mapping.json'
      plan: './tests/unit'
      mocha:
         options:
          ui: 'bdd'
          reporter: 'spec'

      apply: './tests/e2e'
      testcafe:
      screenshot:
        enabled: true
        take-on-fail: true
        path: './tests/e2e/screenshots'
      browsers:
        - firefox --headless

• Add a possibility to configure browser type and mode for e2e execution (e.x. - firefox --headless)
• If it is not defined use the default recink parameters for browser and its mode

[ddimitrioglo]

@eistrati please move mapping: './tests/mapping.json' functionality into a new issue, it should be fixed beside with git diff issue (in Jenkins)

[eistrati]

https://github.com/MitocGroup/recink/issues/122

[eistrati]

@ddimitrioglo I've tested and merged the PR. Nevertheless, this issue is not over yet, as testing terraform-demo didn't work (no errors, but didn't pass the tests either).

[eistrati]

@ddimitrioglo If terraform plan or terraform apply is skipped, I this we should skip corresponding testing step. What do you think?

[eistrati]

@ddimitrioglo Also, if testing step fails, the expected behavior is that whole workflow should stop. It doesn't happen... Here below what happens in reality:

[TERRAFORM]
### 'ec2_dev' returned below output while executing 'terraform plan'

[...]

Terraform show log assertion
    1) Test terraform show log should fail if contains "new resource required" keyword
    2) Test terraform show log should fail if contains "forces new resource" keyword

  0 passing (19ms)
  2 failing

[...]

[TERRAFORM]
### 'ec2_dev' skipped executing 'terraform apply'

Running tests in:
 - HeadlessChrome 0.0.0 / Mac OS X 10.13.2

[...]

[TERRAFORM]
### 'ec2_dev' skipped executing 'terraform destroy'

🚀  Done.

[avozicov]

Test cases with status for recink run terraform command

e2e component:

1. Specify test path including test type (e.x. ./tests/e2e)

   • expected to run all e2e tests from e2e dir = PASSED

2. specify test path including resource directory (e.x. ./tests/e2e/suite/services/rds)

   • expected to run all e2e tests from rds dir = PASSED

3. specify test path including e2e tests for particular resource or service (e.x. ./tests/e2e/suite/services/rds.e2e.js)

   • expected to run particular e2e test for rds service from suite/services directory = FAILED (test execution is aborted with the error message, it is required to place even one test into a directory to be able to trigger targeted execution)

4. specify test path including suite directory (e.x. ./tests/e2e/suite/)

   • expected to run all e2e tests from suite directory = PASSED

5. specify test path including test directory and link it to the apply state under test config in .recink.yml (e.x. apply: “./tests/“)

   • expected to run only e2e tests from suite directory and skip unit tests in case they are placed in the same directory = PASSED

6. specify apply state under terraform config in .recink.yml and set it to false specify valid e2e test path under test config for the same module

   • expected to not run e2e tests in case when apply is set to false = FAILED (tests are still executing if test path is specified under test configuration for apply statement)

7. specify an invalid tests path including non-existent directory (e.x. ./tests/e2e/non-existent-directory/suite)

   • expected path validation and user friendly message that tests execution is aborted because of invalid path) = FAILED (e2e tests are not triggered and there is no any additional information)

8. specify browser type and mode under e2e component in .recink.yml (e.x. Firefox --headless)

   • expected execute e2e tests on Firefox = FAILED (e2e tests are running on Chrome with --headless mode as it is a default recink configuration for e2e component)

unit component:

• described behavior from e2e component is the same for unit test component (except N8)

[avozicov]

@eistrati @ddimitrioglo I would suggest to keep the status "in Review" for the following ticket and lets discuss the described use cases and requirements on our next meeting.

[ddimitrioglo]

3. We can switch to RegExp instead of full path in plan: '/path/to/units/'
4. Looks a valid suggestion
5. We can tell the user "that no tests found in the path '/path/to/units/'"
6. Have no idea for now...

@eistrati please review and provide your feedback

[avozicov]

@eistrati @ddimitrioglo FYI I have added separate tickets for the Failed tests https://github.com/MitocGroup/recink/issues/128 https://github.com/MitocGroup/recink/issues/127 https://github.com/MitocGroup/recink/issues/129 https://github.com/MitocGroup/recink/issues/130

[eistrati]

Checked.