CumulativeMs < (BeforeAllMs+BeforeEachMs+TestOnlyMs+AfterEachMs+AfterAllMs) ?

[vincent-fuchs]

I was expecting CumulativeMs to be the sum of all the other numbers, but I see that regularly it's not the case and it's sometimes (largely) inferior :

how is the CumulativeMs number computed ?

[grzegorz-aniol]

@vincent-fuchs it's quite interesting, there should not be such differences. Cumulative values is a difference between two timestamps, when test class was started and ended. To be more precise, the extension uses JUnit5 callbacks BeforeAllCallback and AfterAllCallback. For sure it shouldn't be higher than sum of individual metrics (before, test only, after). Maybe there is a bug. I will try to reproduce but if you can provide any simple class with this issue that would be nice.

[grzegorz-aniol]

Do you use a mixture of test classes with "without-nested" and with nested classes maybe? I didn't test this type of scenario yet.

[grzegorz-aniol]

I've just released 1.1, you may try to reproduce it, as it solves #2

[vincent-fuchs]

some findings on my side :

• for tests that use @Nested , we get one row with only CumulativeMs for the top level, then one detailed row for each nesting level :

The top level number is not the sum of the nested levels though.

• probably something that is not supported yet, that gives strange numbers : some of my tests extend org.junit.jupiter.api.extension.BeforeAllCallback to perform some set up at a global level. is this supported ? it could explain the discrepancies I observe.

[grzegorz-aniol]

Can it be your custom extension is declared before BenchmarkExtension ? E.g.

@ExtendWith(YourCustomExtension.class)
@ExtendWith(BenchmarkExtension.class)
class AbcTest {
...
}

In that can it may not detect/measure correctly all callbacks, as it needs to overlap all others, and be used as first extension in a chain by JUnit5. Also I found that AfterAll metric, for enclosing class, may be measured incorrectly when nested test is used. I will fix this, but I think it's not your case, as you don't use it.

[grzegorz-aniol]

Also, I found problem with Nested tests and how timestamps are collected in that case, I will work on it.

[grzegorz-aniol]

I improved the way how junit5-watcher handles nested tests. In addition, README contains more details about metrics, how they are calculated and how nested class metrics imply enclosing class metrics

[grzegorz-aniol]

Fix will be included in the next release, planned this week.

[grzegorz-aniol]

Released in 1.2