TCK: There is no easy way to exclude a single test method

[mkouba]

It's possible to instruct maven-surefire-plugin to exclude some test classes. However, sometimes it's useful to only exclude some of the methods. E.g. there is a fix for HistogramTest - https://github.com/eclipse/microprofile-metrics/commit/9fc777c7ac2e0ed72aa517b0bad2d604496827da. We have to exclude the whole test class instead of HistogramTest.testMetricRegistry().

[jmartisk]

I don't understand what you expect MP Metrics should do about this. It is possible to do mvn test '-Dtest=!HistogramTest#testMetricRegistry' to exclude one method

[mkouba]

Well, other specs provide a way to exclude a single test method easily. For example, testng based TCKs allow the implementor to specify an XML file that contains exclusions with additional description. See for example https://github.com/smallrye/smallrye-fault-tolerance/blob/master/testsuite/tck/src/test/tck-suite.xml#L32-L37.

Metrics TCK is using junit and so you could either exclude the whole test class (using Maven surefire config) or the exclude the method using the test system property. Neither of these approaches is very elegant and easily maintainable.

[jmartisk]

In that case we might have to migrate to TestNG, as I don't think introducing a home-baked XML suite solution for JUnit would be a good idea. Related issue about moving to TestNG: https://github.com/eclipse/microprofile-metrics/issues/246

@kenfinnigan, is there any outcome from the Architectural Group regarding the choice of testing frameworks?

[kenfinnigan]

There isn't to date as there was not enough attendance in the last call.

Hopefully this month we can discuss it