Errors incorrectly detected when failures="0"

steve-taylor commented 2 years ago

Consider the following xunit output:

<testsuite name="Testem Tests" tests="18" skipped="0" failures="0" timestamp="Mon Nov 08 2021 10:21:17 GMT+1100 (Australian Eastern Daylight Time)" time="8.652">
    <!-- Redacted -->
</testsuite>

This is being reported as an error, which I've traced to detection.sh line 19.

I noticed there's inconsistent usage of the -L option of grep. This is important because the -L option not only suppresses output (replacing it with the name of the file), but also inverts the match.

Interestingly, when I ran this locally on macOS to reproduce the issue, I couldn't reproduce it. However, I was able to reproduce it in Linux. It seems that grep -L always returns 0 on macOS, whereas it returns 1 on Linux when a match is found (i.e. opposite of what grep -l would do).

The pseudocode of this snippet:

if echo "$suite_line" | grep "failures=\"0\"";then
    if echo "$suite_line" | grep -L "errors=\"[1-9]\d*\"";then
        echo "Error found in $file: $suite_line"
        return 1
    else
        return 0
    fi
elif echo "$suite_line" | grep -L "errors=\"0\"";then
    if echo "$suite_line" | grep -L "failures=\"[1-9]\d*\"";then
        echo "Error found in $file: $suite_line"
        return 1
    else
        return 0
    fi
else
    echo "Failure found in $file: $suite_line"
    return 1
fi

appears to be the following:

If the test report contains failures="0", then
- If the test report doesn't contain errors="1+", then one or more tests failed.
- Otherwise, all tests passed.
Otherwise, if the test report doesn't contain errors="0", then
- If the test report doesn't contain failures="1+", then one or more tests failed.
- Otherwise, all tests passed.
Otherwise, one or more tests failed.

More concisely, failure occurs if one of the following is true:

The test report contains failures="0" and doesn't contain errors="1+";
The test report contains neither errors="0" nor failures="1+"; or
The test report doesn't contain failures="0" and does contain errors="0".

This is what I think is needed:

Fail if the test report contains failures="1+";
Fail if the test report contains errors="1+"; or
Fail if the test report is missing the failures attribute (could be a malformed test report).

In which case, find_fail should become:

find_fail() {
    target=$1
    file=$2
    suite_line=$(cat "$file" | grep "$target")

    if echo "$suite_line" | grep -q -E "failures=\"[1-9]\\d*\""; then
      echo "Failure(s) found in $file: $suite_line"
      return 1
    elif echo "$suite_line" | grep -q -E "errors=\"[1-9]\\d*\""; then
      echo "Error(s) found in $file: $suite_line"
      return 1
    elif echo "$suite_line" | grep -qv -E "failures=\"0\""; then
      echo "Test report may be malformed: No errors were detected, but the failures attribute is missing from the testsuite element."
      return 1
    fi
}

As a bonus, it also works on macOS.