Missing `assert_stderr` and similar assertions

[jbriales]

The run command in bats-core has the ability to keep stderr-related output into separate variables:

Use --separate-stderr to split stdout and stderr into $output/$stderr and ${lines[@]}/${stderr_lines[@]}

as per https://bats-core.readthedocs.io/en/latest/writing-tests.html#run-test-other-commands

Should we have assertion helpers that apply to stderr only?

[martin-schulze-vireso]

Since this is a relatively recent addition to bats-core, I would be cautious to add a feature that relies on a recent addition. We would need some kind of guard to avoid breaking suites by using the wrong bats version. I also dislike the amount of code duplication that entails. Would prefixing assert_output with output="$stderr" be a bad option?

[jbriales]

I think the proposed workaround is very spot on and indeed works nicely and unblocks here. I think with this alternative, it's not worth cluttering the code and duplicating functionalities as you mentioned. Closing this. Thanks!

[viathor]

@jbriales @martin-schulze-vireso I'd like to revisit this.

I agree there is a viable workaround. However, normally bats tests tend to look very clean and elegant and the workaround is rather jarring. Also, --separate-stderr is no longer as recent as it was back when this was first filed.

Code duplication could be avoided by implementing this as a flag, say -r, --stderr added to both assert_output and assert_line.

WDYT?

[jbriales]

I agree on bats being generally quite clean and this workaround getting dealing with stderr checks less seamless or "native".

I'm not sure about the impl details, but if we went for extending, I'd find having assert_stderr more discoverable than assert_output --stderr.

[martin-schulze-vireso]

I agree on bats being generally quite clean and this workaround getting dealing with stderr checks less seamless or "native".

I'm not sure about the impl details, but if we went for extending, I'd find having assert_stderr more discoverable than assert_output --stderr.

I'd Vote for this as well. The existing flag parser in assert_output Just assumes flags it does not know to be the test expectation. This means you will get a somewhat misleading error message in older versions.

[viathor]

Yeah, assert_stderr sounds good, too. I suppose the counterpart for assert_line would be something like assert_stderr_line or assert_line_stderr?

[jbriales]

Yeah, assert_stderr sounds good, too. I suppose the counterpart for assert_line would be something like assert_stderr_line or assert_line_stderr?

Because it would refer to a check in a specific line vs checking against the full output, I'd say assert_stderr and assert_stderr_line would be the most intuitive pair?

[martin-schulze-vireso]

Because it would refer to a check in a specific line vs checking against the full output, I'd say assert_stderr and assert_stderr_line would be the most intuitive pair?

assert_stderr and assert_stderr_line sounds good to me.

[viathor]

Sounds like there is agreement that this should be done (and even how it should be done). Perhaps it could be reopened? :-)

[hgl]

Another case against output="$stderr" is that if you use shellcheck, it's going to complain stderr not being defined, so you'd have to either use ${stderr?} or add a shellcheck disable comment.

With assert_stderr, there is no need to jump through such hoops.

[mh182]

Is someone working on this? I also stumbled on this problem. The first thing I did was searching for assert_stderr. Consistency is a powerful concept.