search

WASdev / standards.jsr352.tck

Home of the TCK for the Java Batch standard (JSR 352, Batch Applications for the Java Platform). #JavaEE7
Apache License 2.0
1 stars 9 forks source link

Add annotation(s)? linking individual tests to spec subsections in organized manner. #6

Open scottkurz opened 9 years ago

scottkurz commented 9 years ago

E.g. see ML msg: https://java.net/projects/jbatch/lists/public/archive/2015-09/message/7

scottkurz commented 9 years ago

Started by adding annotations here.

Next step is to use them by annotating some tests.

scottkurz commented 9 years ago

Not going to be interesting until we have more tests annotated, but besides applying the annotations,another questions is how to build reports from them (whatever form that's in.. standalone html. . or tied to Javadoc somehow???

jvfullam commented 8 years ago

Annotations were updated here.

jvfullam commented 8 years ago

Some thoughts, now that it's been a few weeks since we remodeled the annotations and now that we have a few of the tests annotated:

So, how can we make them shorter?

Let's back up for a moment and think about what we're really trying to accomplish with the annotations: The whole point of the TCK is to ensure that a given java batch implementation is compliant with the JSR 352 specification. In order to be compliant with the spec, an implementation must behave in all of the ways defined (or asserted) by the spec.

So what we're really looking for is a way to see - at a glance - whether or not we have a test for every behavior defined by the spec (we call these "assertions"); we want to know what assertions we still need to add tests for.

In order to do this, it follows that the spec itself should maintain a list of assertions for each of its subsections. Then, a test can simply reference the assertion(s) that it's testing.

In the meantime (while we do not yet have a list of assertions to reference) maybe the next best thing is to simply reference the spec subsection where the behavior is described (a subsection contains 0 to many assertions). Then, at some later point in time when the assertions are labeled, it will be easy enough to replace the subsection reference with the more specific assertion reference.

jvfullam commented 8 years ago

On the same topic: we currently have some tests that verify behaviors that are not asserted in a single spec subsection, but rather are extrapolated from a combination of subsection assertions. In these cases, we do not necessarily have to give the test an annotation. Though I think the real issue in these situations is that the spec itself should be made more explicit, so that it offers a definitive, single point of reference for some of these more complicated behaviors that we want to test.