KStocky / ShaderTestFramework

An automation testing framework for testing shader code
MIT License
20 stars 1 forks source link

Assert on tests that use sections as if they were C++ tests #62

Open KStocky opened 5 months ago

KStocky commented 5 months ago

Is your feature request related to a problem? Please describe. Now that we have named sections, we could be asserting on the test result information as if we had written the tests in C++. By this, I mean that if you run the tests in C++ TestMate then a section on the HLSL side, would be displayed as a section in the C++ Test Mate test runner. I think this would be really, damn cool

Describe the solution you'd like This will be a large refactor. At the minute we just format failed assertions. But to support this we would instead have to iterate through every section and assert in the section if there was an assertion in it. Associating a list of assertions to a section could be interesting. At the minute the relationship is the other way around allowing for a 1 to 1 relationship. This will be a 1 to many relationship. The nice thing is that we would be able to rip out operator<< for TestRunResults. We would no longer have to format our errors. Just the individual asserts. Catch will handle the section formatting.

Additional context Here is a screenshot of what I mean image

Each nested section gets reported as such in C++ Test Mate.

KStocky commented 1 month ago

It's worth noting that this isn't really possible since #124. We could do it but we wouldn't have the string information because we now only compile strings on a failure run. This is to improve compilation times. We can still implement this because test writers can enable strings at all times if they change the default. We might want to introduce shader caching for this though. Compiling shaders with strings is very expensive.