Match output of catch2 when listing tests with default reporter

CrustyAuklet commented 2 months ago

My team uses Clion and while transitioning our libraries to snitch from Catch2 I noticed CLion choking on the tests. Clion has built in Catch2 support, and in the readme snitch claims to support Clion via this.

I modified the snitch main function to write the value of argv to a file, and saw that clion calls the test binary twice:

before running tests with --list-tests --order lex ~[.]
run the tests with -r xml -d yes --order lex

I compared the output with the list tests command between the latest Catch2 and snitch. I copied a representation of the difference at the bottom of this description. But the key differences are:

tests aren't ordered
no header text "Matching test cases:"
no final "132 matching test cases"
test names aren't indented
no tags output for each test

Of these items Clion at least only seems to care about 2, 4, and 5.

Adding the test count at the end doesn't matter to Clion and would be a much larger change. I have two branches in my repo here I attempted it and it was either:

Static variable in the default reported struct. Which isn't very nice.
change default reporter to a struct with a counter variable as member. A large change that I didn't actually get to work.

Catch2

$ unit-tests-catch2.exe --list-tests --order lex ~[.]
Matching test cases:
  Asserts and contracts
      [assert]
  COBS framing
      [cobs][ranges]
  COBS unframing
      [cobs][ranges]
  CRC Algorithm
  GPS <-> UTC Conversions
  block_allocator
      [blocks][memory][pool]
  block_allocator custom alignment
      [blocks][memory][pool]
  inplace_function
      [functional][larid]
  is_scoped_enum
      [c++23][type_traits]
  memory_pool_external
      [memory][pool]
  memory_resource
      [larid][memory_resource]
  ........... many more tests I won't waste space on .......
132 matching test cases

No test listed

$ unit-tests-catch2.exe --list-tests --order lex [.]
Matching test cases:
0 matching test cases

Snitch

$ unit-tests-snitch.exe --list-tests --order lex ~[.]
warning: unknown command line argument '--order'
COBS framing
COBS unframing
basic_netbuf
empty ispanstream
non-empty ispanstream
empty istreambuf_iterator
empty ostreambuf_iterator
memory_resource
block_allocator
block_allocator custom alignment
memory_pool_external

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.06%. Comparing base (86d18cb) to head (ade8932).

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/snitch-org/snitch/pull/166/graphs/tree.svg?width=650&height=150&src=pr&token=X422DE81PN&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org)](https://app.codecov.io/gh/snitch-org/snitch/pull/166?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org) ```diff @@ Coverage Diff @@ ## main #166 +/- ## ========================================== + Coverage 94.05% 94.06% +0.01% ========================================== Files 27 27 Lines 1698 1701 +3 ========================================== + Hits 1597 1600 +3 Misses 101 101 ``` | [Files](https://app.codecov.io/gh/snitch-org/snitch/pull/166?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org) | Coverage Δ | | |---|---|---| | [src/snitch\_reporter\_console.cpp](https://app.codecov.io/gh/snitch-org/snitch/pull/166?src=pr&el=tree&filepath=src%2Fsnitch_reporter_console.cpp&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org#diff-c3JjL3NuaXRjaF9yZXBvcnRlcl9jb25zb2xlLmNwcA==) | `98.92% <100.00%> (+0.03%)` | :arrow_up: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/snitch-org/snitch/pull/166?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/snitch-org/snitch/pull/166?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org). Last update [86d18cb...ade8932](https://app.codecov.io/gh/snitch-org/snitch/pull/166?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=snitch-org).

cschreib commented 2 months ago

Thanks for reporting this. We can merge this PR without the "X matching test cases", and leave that for a subsequent PR.

change default reporter to a struct with a counter variable as member. A large change that I didn't actually get to work.

This would be the correct approach. Do you want to pursue it, or would you rather I take it from here? If you want to keep at it, feel free to open a draft PR and we can provide some feedback along the way.

CrustyAuklet commented 2 months ago

This would be the correct approach. Do you want to pursue it, or would you rather I take it from here? If you want to keep at it, feel free to open a draft PR and we can provide some feedback along the way.

I will open a draft PR with what I have. I think it is 95% done and I will point out where I got stuck in the PR.

snitch-org / snitch