CLI filtering/matching test case sections

[jeremyong]

First of all, thanks a lot for this library!

I was curious if there was a mechanism to list/match sections contained within test cases. Sections are a very convenient way to handle test setup/teardown, but the unfortunate drawback is that sections can't be run in isolation as far as I can tell.

[cschreib]

Hi! There is currently no way to do this. Filtering a test case to only run certain sections would be possible, without too much cost, but figuring out the filtering syntax and how it meshes with the existing test case filtering will require some thinking.

As for listing sections, that would also be possible but with a big caveat: it would require actually executing all the test code. Indeed, sections aren't known when entering a test case, they are discovered as the test code executes and encounters SECTION blocks. To bypass this, we'd need some sort of introspection (reflexion) on the code of the test case, and that's not likely to happen anytime soon! In the meantime, it could be implemented in the naive way, executing the code, but that's going to be too slow for most applications, and also risks executing failing REQUIRE*() checks that fail and abort... I suspect that's not going to be practical.

I know it's tempting to have one big test case with lots of sections in it :) But my advice here is, if you start to feel the need to execute only one section, then it's probably a sign that you should create a different test case. You can factor out the common code the old fashioned way, with functions and classes.

[jeremyong]

For the filtering syntax, I was thinking that something simple would be to use a . access pattern. For example, a test case named Foo with sections S0, S1, and S2 could be filtered using Foo.S0 to run only the section S0 of test case Foo. This has the drawback of possibly colliding with tests that have a ., but given that . is used as a special symbol in the case of tags, it's not unreasonable to expect that . behaves differently in the test naming case also. I feel like . subsection handling is far too natural to consider anything more complicated (although I could be persuaded of course, this is just my gut reaction). You can imagine that *-based pattern matching could apply to the test case prefix in addition to the section suffix (if present).

Agreed that providing test section listing support might be too much trouble than what it's worth.

[cschreib]

For reference, Catch2 does this with a separate CLI option --section .... If this is implemented in snitch, we will want to follow the same CLI API to benefit from IDE integrations etc.