search

microsoft / xunit-performance

Provides extensions over xUnit to author performance tests.
MIT License
188 stars 55 forks source link

When all tests fail we do not make an output XML #193

Closed DrewScoggins closed 3 years ago

DrewScoggins commented 7 years ago

If all of the tests that are run inside of a DLL fail we do not create an output XML. Is this the intended behavior? It seems like we should at least make an empty XML file with no results in it.

brianrob commented 7 years ago

I would expect an XML file as well. @DrewScoggins is this specific to xunit.performance.run or is this also present when using xunit.performance.api?

jorive commented 7 years ago

Why would you expect an output file with results when there is a failure? That behavior does not seem correct to me.

brianrob commented 7 years ago

Let me clarify. If all of the tests are run, even if they fail, I would expect to have a results file. The results file will be full of failures, but it will exist. If on the other hand, the harness throws or crashes, then I would not expect to see an XML file, and this is likely out of your control anyway. What do you think @jorive?

jorive commented 7 years ago

@brianrob Maybe I am missing something, but it does not make sense to have a file if all test failed. You will get an etl profile of the run, but you do not have statistics compute, print or output, or data to upload to BenchView (no raw data). What would those files have? The tests names?

brianrob commented 3 years ago

Closing prior to archive.