search

cdevents / spec

A common specification for Continuous Delivery events
Apache License 2.0
129 stars 22 forks source link

Should testSuiteRun be a required field in testCaseRun #166

Open dan-han-101 opened 11 months ago

dan-han-101 commented 11 months ago

testCaseRun currently contains an optional field for testSuiteRun.

For our use cases, we always have a mapping between a test case and suite, so it would be good for us to just make this a required field.

Were there any use cases where test cases are not expected to be part of a test suite? Thanks!

e-backmark-ericsson commented 11 months ago

The discussions have been around whether there always is a test suite to refer to or not. Some users might not see the need for a test suite at all. Some users might see the pipeline step execution as the test suite and would then instead like to refer the test case executions to that taskRun executions, which could potentially be made possible through #104 .

I'm not a strong opponent of requiring a testSuiteRun to always be sent and thus required to reference from the testCaseRun though.

afrittoli commented 10 months ago

Users that do not rely on that grouping of tests would not know what to provide in terms of testSuiteRun - I guess they could always provide the testRun itself as testSuiteRun in case they didn't have one, but that seems unnecessary.

@dan-han-101 I would like to better understand the benefit you would gain from having testSuiteRun mandatory. Does it solves a specific issue you're facing when using test events?

dan-han-101 commented 10 months ago

The background / context for this question stems from the use of junit xml test reports.

Many test runners, such as pytest and jenkins report test reports in this common format.

Moreover, this junit format nests test cases under test suites.

Given this common pattern of having all test cases link to some test suite, I thought it might make sense for CDEvents spec to also require this mapping.