Bedrock Integration Tests tracking with junit logs from terratest

[NathanielRose]

As an: Operator

I want: Visibility to unit & integration tests over time in devops pipeline,

So that: I can monitor the success rate, common failure errors and capture logs of failed builds.

Describe the solution you'd like: Using a JUnit report with the terratest_log_parser, which outputs the Junit report, to be captured in our pipeline.

Acceptance Criteria:

• [x] All Integration test produce Junit report
• [x] Junit report is integrated in Devops Pipeline framework

Additional context:

• https://github.com/gruntwork-io/terratest/issues/350
• https://github.com/gruntwork-io/terratest#debugging-interleaved-test-output
• https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
• Sample: https://dev.azure.com/epicstuff/bedrock/_test/analytics?definitionId=128&contextType=build

Does this require updates to documentation?: Yes

[andrebriggs]

@NathanielRose the goal is is to have visibility about test failure. We don't want to have silent failures. Can we simply just check the exit code of terra test and fail the entire pipeline if we get a non-zero exit code?

Might be a better short term solution than letting this linger

[andrebriggs]

Kicking this out of SPK

[andrebriggs]

An approach that may work

We can run each test like the following:

$ go test -v -run TestIT_Bedrock_AzureSimple_Test -timeout 99999s |  tee terratest-ci.log     

Then use terra test_log_parser

$ terratest_log_parser --testlog terratest-ci.log --outputdir test_xml

Which produces a test_xml folder

Then add this yaml task

  - task: PublishTestResults@2
      inputs:
        testResultsFiles: '$(Agent.BuildDirectory)/s/test_xml/report.xml'
        mergeTestResults: false
        failTaskOnFailedTests: true

Since we may have multiple or conditional test runs we may need to merge multiple test results into one xml file.

We can maybe follow the approach here: https://gist.github.com/cgoldberg/4320815

cc @NathanielRose

[NathanielRose]

Issues with logging recorded here - https://github.com/gruntwork-io/terratest/issues/524