search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
345 stars 343 forks source link

Tap runner hides failures #6007

Closed richtja closed 2 months ago

richtja commented 3 months ago

Describe the bug If one of the tests in tap output is skipped before some failure, avocado will mark the whole output as skip instead of fail and users are not notified about the failure.

Steps to reproduce TAP output generated by tap test stored in example.tap:

TAP version 13

# MUST be treated as a SKIP test
ok 1 - must be skipped test # SKIP

not ok 2 - actual failure
ok 3 - passed test

bash script for reading this file as test.sh:

#!/bin/bash - 

cat example.tap

JSON for running the tap test as test.json:

[
    {"kind": "tap", "uri": "./test.sh"}
]

Avocado command:

$ avocado run test.json

Expected behavior A clear and concise description of what you expected to happen.

JOB ID     : f9305c1fa08da1d0a2e0a668e76905e23182e46a
JOB LOG    : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/job.log
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: STARTED
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: FAIL
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/results.html
JOB TIME   : 3.51 s

Current behavior What you are getting?

JOB ID     : f9305c1fa08da1d0a2e0a668e76905e23182e46a
JOB LOG    : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/job.log
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: STARTED
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: SKIP
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/results.html
JOB TIME   : 3.51 s

System information (please complete the following information):

richtja commented 2 months ago

Resolved in #6010