Bnd unit test and bundle test results go into different folders

[timothyjward]

When building using Gradle we get different output folders for test results:

Unit tests: generated/test-results/TEST-XXX.xml

Bundle tests: generated/test-reports/TEST-XXX.xml

[bjhargrave]

You don't specify a problem here.

Gradle's java plugin uses test-results. So this is what people will expect with gradle test task.

bnd uses test-reports. So this is what people will expect with bnd's OSGi junit tests (what the gradle check tasks runs.)

I am not sure that it is necessary or a great idea to use the same folder for the test output. What if the same test class is run in junit and OSGi junit settings? Then, for that test class, the output of one setting will overwrite the other. Why do you think they should use the same folder? CI systems can easily be configured to aggregate the test results in both folders: **/generated/test-*/TEST-*.xml

[timothyjward]

The problem as I see it is inconsistency, rather than anything else. The difference seemed arbitrary, and most likely to be an accident. If you're telling me that it's not an accident then that's ok, but it is still surprising. Bundle tests run using JUnit, and have always been made to feel as "unit test like" as possible. I would therefore expect them to end up in the same place as other unit test output. Other bnd-based plugins have also tried to follow the expected layout for that build tool, for example the Maven Bundle Plugin puts its generated data into /target not /generated.

Restrictions around running the same test both inside and outside seems like a fairly tenuous argument. I've not heard of anyone trying to do that, and it would be really bad testing practice to amalgamate all of those tests into one class. Also the unit test classes live in /test (or /src/test/java) and the bundle tests in /src/ (or src/main/java). To get a clash in test case names would mean that someone had created duplicate classes on their test class path already. This is a scenario that will always be fraught with difficulties.

The reason that I raised this bug is that these sorts of details matter when trying to set up an environment, and when trying to integrate with other tools. Existing Gradle users have to know to adjust their CI configuration to handle bnd being different, making it yet one more barrier to entry. Yes it's solvable, but it still bit me, and it still took time to work out why the bundle tests weren't being picked up. I don't see any reason not to make this layout consistent when building using Gradle and avoid the problem entirely.

[bjhargrave]

So what solution do you propose? Change the default in bnd to match the gradle java plugin's default? Or change the default in the bnd gradle plugin to match bnd's default? Given that we have already shipped bnd and the bnd gradle plugin, someone will be affected by the next release of bnd and the bnd gradle plugin.

[timothyjward]

My proposal would be to have the bnd gradle plugin use the same junit xml output location as the java plugin by default.

Existing users (early adopters like me) will have a small migration step in their CI builds to remove unnecessary JUnit output filters, but won't actually see failures. This migration will be a deliberate process; the version of my plugin only changes when I choose, not when I upgrade bndtools.

People who aren't using the plugin yet won't hit issues at all, and CI configuration templates for Gradle will work out of the box, rather than needing customisation for bnd.

[bjhargrave]

My proposal would be to have the bnd gradle plugin use the same junit xml output location as the java plugin by default.

Sorry, but I am not clear on your meaning. Do you mean that the bnd ProjectTester class changes to use test-results like the gradle java plugin. So the check task outputs to the test-results folder used by the test task? Or do you mean the bnd gradle plugin configure the test task to use the test-reports folder currently used by bnd ProjectTester class and thus the check task?

I think you are saying the former, but I am unclear. I am trying to figure out which code changes.

[fhuberts]

On 17/12/14 17:50, Tim Ward wrote:

My proposal would be to have the bnd gradle plugin use the same junit xml output location as the java plugin by default.

+1

[timothyjward]

Sorry, but I am not clear on your meaning.

Looking back, it could have been clearer. I was trying to suggest that the bnd gradle plugin code change to configure the test task to use a test-reports folder named test-results.

Changing the ProjectTester would also be fine with me, although that appears to be a change for all of bnd, rather than just Gradle users.

[bjhargrave]

Looking back, it could have been clearer. I was trying to suggest that the bnd gradle plugin code change to configure the test task to use a test-reports folder named test-results.

LOL. I think you mean: configure the test task to output into a folder named test-reports which is the folder name used by ProjectTester. So this would actually mean configuring the gradle project so that the project property testResultsDirNameis set to the value of the bnd project's test-reports property. For example:

testResultsDirName = bnd('test-reports', 'test-reports')

[fhuberts]

No, he means that the OSGi tests results are put in the same dir as the regular unit tests, following the Gradle convention (however you want to configure that)

[bjhargrave]

No, he means that the OSGi tests results are put in the same dir as the regular unit test, following the Gradle convention (however you want to configure that)

In that case, for consistency, you need to change ProjectTester so the same default output folder is used when running gradle check or inside Eclipse.

Since the output folder of ProjectTester is configurable via the bnd project test-reports property, it actually make more sense to configure the gradle test task to use the bnd project test-reports property. Then the bnd project property can set the folder name for all cases.

[timothyjward]

Perhaps I am getting confused - It looks as though this line sets the report directory to a configurable value with the property name "test-reports", with a default value of "test-reports". My understanding was that the bnd gradle plugin could simply set this property (unless it already exists) to the value "test-results" and have the JUnit output appear in that location.

Ok - this came in while I was typing

Since the output folder of ProjectTester is configurable via the bnd project test-reports property, it actually make more sense to configure the gradle test task to use the bnd project test-reports property. Then the bnd project property can set the folder name for all cases.

This sounds fine, there is then the issue of whether the default output location changes. This could be done in ProjectTester, or as the bndtools default build tool is now Gradle an update could be made to the default master.bnd. Either would ensure that the XML output location is suitably consistent for JUnit and Bundle tests. The former has the benefit of being more consistent "by default" but does have the drawback of being more disruptive to existing users.

[bjhargrave]

The former has the benefit of being more consistent "by default" but does have the drawback of being more disruptive to existing users.

Whose existing users? bnd which has always used test-reports? Or the gradle plugin users who now have different folders for the test and check tasks? :-)

Given that there is a (poorly documented) test-reports bnd project property which can be used to set the folder to hold the ProjectTester (OSGi Junit) output, I think we should use that bnd project property to establish the output location for both Junit and OSGi Junit output.