search

snitch-org / snitch

Lightweight C++20 testing framework.
Boost Software License 1.0
252 stars 7 forks source link

Draft: Feat/catch2 compat #169

Open CrustyAuklet opened 2 months ago

CrustyAuklet commented 2 months ago

Very rought draft, but I wanted to open it so others can look and comment.

This is work to make the Catch2 XML output match up with Catch2. Like the other open MRs I notices this when integrating with my teams IDE and (more importantly) CI test reporting systems.

Without these changes CI and IDEs work ok, but failures in a section just fail the whole test. When we were using Catch2 a failure would be specific to a section allowing us to quickly drill down to the problem. So looking into the difference it is just the lack of section output in the "high" verbosity mode. Catch2 outputs section XML data by default.

It was fairly easy to make it work with basic testing, pass and fail. There are more complex edge cases around skips, nested sections, exceptions being thrown, and different verbosity levels. I am still ironing those out and need to make sure there is testing for it all.

As I have time this week i will add some examples here, and some pictures to show the different in CI/IDEs.

cschreib commented 2 months ago

Thanks for spotting this. I could have sworn I got the output for sections to match Catch2 when I last tested this; but perhaps I had looked at a too simple case and missed the bigger picture.

As I have time this week i will add some examples here, and some pictures to show the different in CI/IDEs.

It would be very interesting to compare the actual XML output between the two in a real-world case; the test project I use for testing and benchmarks doesn't actually use sections.

CrustyAuklet commented 2 months ago

thanks for the feedback, I will look it over in depth this coming week.

I've been working on making tests and comparing Catch2 to Snitch output in a repo on my account here: https://github.com/CrustyAuklet/snitch-xml-output. So far it's pretty close but I will have to add some more variations, especially to cover the exception stuff.

cschreib commented 1 month ago

Hi there! Just wondering if this was still on your mind. Let me know if not, and I can take over.

CrustyAuklet commented 4 weeks ago

It is, I have just had a ton on my plate. I am going to try to re-familiarize myself with it this afternoon and see if have any questions on your feedback.

I won't be offended at all if you think it's an easy fix for you to do and you want to get it done 😄

CrustyAuklet commented 3 weeks ago

fixed some of the small easy stuff and rebased, but spent a lot of time running some example tests in that other repo with exceptions. I think most of this code needs to be moved to the registry like you said.