Bug: Testscript passes when it should fail

[cassamajor]

Here is the Main function for match:

func Main() int {
    m, err := NewMatcher(
        WithSearchTextFromArgs(os.Args[1:]),
    )
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return 1
    }
    m.PrintMatchingLines()
    return 0
}

Let's remove the ability to process arguments

func Main() int {
    m, err := NewMatcher()
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return 1
    }
    m.PrintMatchingLines()
    return 0
}

Executing the tests will still pass.

---

Let's also modify matches_lines_from_stdin.txtar to add one additional line that should match

stdin three_matching_lines.txt
exec match MAGIC
stdout -count=1 '^this line is MAGIC\n'
stdout -count=1 '^MAGIC is also on this line\n'
stdout -count=1 '^and MAGIC is here too\n'

-- three_matching_lines.txt --
this won't match
this line is MAGIC

MAGIC is also on this line
but not this one
and MAGIC is here too
MAGIC is a lie

Tests still pass, even though we haven't added a stdout directive to match the new line.

[bitfield]

Good spotting, @cassamajor! Have you read The Power of Go: Tests? You're certainly thinking like someone who has.

Can you suggest a suitable change to catch this bug?

[cassamajor]

I haven't, but it's on my list to read!

Unfortunately, I am not familiar with testscript, and I do not know a suitable fix.

[bitfield]

The problem is that these stdout assertions are not exclusive: that's to say, they specify that the output from the command should contain X, but they don't also say that it shouldn't contain anything else!

We can see from the code in match.go that the matcher's default text is the empty string, and as far as strings.Contains is concerned, every input line contains the empty string!

Therefore, we need to test that the output contains the lines expected to match, but we also need to test, as you've shown, that it doesn't contain lines that shouldn't match.

We can add that with a negative assertion, using the ! character. Can you see how to fix it?