Improve test results reporting

[psss]

There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:

• https://pagure.io/fedora-ci/general/issue/18

Let's update the pipeline according to the agreed result: The main difference is that ansible-playbook should return non-zero status only upon infrastructure errors while tests failures can be detected from the newly defined file results.yml which will have the following format:

results:
- {result: pass, test: shell/smoke}
- {result: fail, test: shell/login}
- {result: error, test: shell/func}

Extended version with links to logs:

results:
- {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log}
- {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log}
- {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}

The expected behaviour of the pipeline is this:

• Report error when ansible returns non-zero (infrastructure error)
• Report error if error appears in results.yml files
• Report test failure if fail appears in results.yml files
• Report pass otherwise

This issue is blocked by implementation of the new functionality in Standard Test Roles which is tracked here:

• https://pagure.io/standard-test-roles/issue/296

[johnbieren]

@bgoncalv This is something we should address in the Jenkinsfile of the upstream-fedora-pipeline, right?

[bgoncalv]

@johnbieren right, I think updating the Jenkinsfile on upstream-fedora-pipeline would be the easiest change. Other option would be after the playbook is executed on "package-test.sh". Where do you think makes more sense to parse the results.yml?

[johnbieren]

@bgoncalv good point, perhaps package-test.sh is best since the error codes are right there and we don't have to rely on the proper error code being propagated through