Closed planetf1 closed 4 years ago
Checked the sonar scans, which report on code coverage .. we can see the OCF tests coverage at https://sonarcloud.io/component_measures?id=odpi_egeria&metric=tests&selected=odpi_egeria%3Aopen-metadata-implementation%2Fframeworks%2Fopen-connector-framework%2Fsrc%2Ftest%2Fjava%2Forg%2Fodpi%2Fopenmetadata%2Fframeworks%2Fconnectors
This is consistent with the maven output -- only 3 tests are executed
Against 705 tests run by gradle ....
I then continued and got to repository-services-apis.
here maven does a lot better, reporting 224 successful, whilst gradle reports 271, 3 failing
AssetCatalog reports 0 tests run in maven, yet in gradle we execute 10, and fail 1 - again, Sonar code coverage confirms none of the code was tested at all
It's worth pointing out that the tests are executing with a default config so tests failing with gradle isn't unexpected or concerning in itself -- it was an experiment and there may be reasons why the environment is failing some tests especially where mockito and other mocking framework is used. However the fact the tests aren't even attempted in our usual build is the key take-away here that needs understanding
Archive of html reports from tests are at https://pastebin.com/0Z4Q0C0n
OUtput from ocf:
Currently our maven surefire plugin is configured to run testcases:
<include>**/*Test.java</include>
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html gives the defaults as
<includes>
<include>**/Test*.java</include>
<include>**/*Test.java</include>
<include>**/*Tests.java</include>
<include>**/*TestCase.java</include>
</includes>
Actually looking for file matches we only see at ffdc/UserNotAuthorizedExceptionTest.java a class which is actually a testNG test, has 3 testcases & does get run ok.
The other tests are in classes Test* -- so in this case our pattern matching is wrong - indeed removing the default override is probably the right step
When we do this we get expected output:
[ERROR] Failures:
[ERROR] TestAssetSchemaAttribute.testClone:270->validateResultObject:139 expected [true] but found [false]
[ERROR] TestAssetSchemaAttribute.testNullObject:182->validateNullObject:162 expected [true] but found [false]
[ERROR] TestAssetSchemaLink.testClone:230->validateResultObject:99 NullPointer
[ERROR] TestSchemaAttribute.testClone:180->validateResultObject:92 expected [true] but found [false]
[ERROR] TestSchemaAttribute.testCloneSchemaElement:189->validateResultObject:92 expected [true] but found [false]
[ERROR] TestSchemaAttribute.testJSON:220 Exception: expected [true] but found [false] expected [true] but found [false]
[ERROR] TestSchemaAttribute.testNullObject:117 expected [true] but found [false]
[ERROR] TestSchemaImplementationQuery.testJSON:129 Exception: expected [true] but found [false] expected [true] but found [false]
[INFO]
[ERROR] Tests run: 705, Failures: 8, Errors: 0, Skipped: 0
So this appears a simple mistake in the surefire plugin definition - easily fixed.
HOWEVER this cannot be merged as-is since unit tests are now failing. Further this will extend to other components too
Suggestions a) One person works on fixing all of these tests en-masse so the fixes and maven change can be checked in together b) Alongside a maven fix we configure maven to i) ignore all tests (too dangerous) ii) ignore selected module tests (hmmm) iii) ignore specific tests using the annotation @Test(enabled = false) c) fix all tests first, then merge the maven change
Although b) is the most changes, it's simple (done in 10s of minutes), and means from now on we will know if any tests fail. It also allows the test fixing to be incremental & distributed. This is my proposal
cc: @mandy-chessell
I've gone through the failing cases and addressed all with one of @Disabled (JUnit5) @Ignore (JUnit4) @Test(enabled=false) (TestNG)
I propose to create a PR with this change, and merge. The owners can then unpick the issue & we can leave this issue open to track - summary once build verified
Issues raised for components
Closing this as build process is now correct (subject to PR merge)
In the maven build we see that only a small number of tests run against the open-connector-framework.
The output from the surefire plugin is:
Running with -X for debug doesn't provide any further hints beyond the execution of 3 tests As part of investigation into some other maven issues, I ran a partial build using gradle. Tests failed
The stats reported were:
inspection shows we have many unit tests in the OCF -- far more than the 3 (!) that surefire runs in our current maven configuration
This needs investigation ... since it may be that we are not executing other tests that we should be....