search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
806 stars 260 forks source link

Not all Unit Tests executed by maven build #3363

Closed planetf1 closed 4 years ago

planetf1 commented 4 years ago

In the maven build we see that only a small number of tests run against the open-connector-framework.

The output from the surefire plugin is:

2020-07-19T09:16:22.8758548Z [main] INFO org.apache.maven.cli.event.ExecutionEventLogger - --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ open-connector-framework ---
2020-07-19T09:16:22.8760707Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - Surefire report directory: /home/vsts/work/1/s/open-metadata-implementation/frameworks/open-connector-framework/target/surefire-reports
2020-07-19T09:16:24.8679025Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - 
2020-07-19T09:16:24.8682909Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - -------------------------------------------------------
2020-07-19T09:16:24.8686235Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin -  T E S T S
2020-07-19T09:16:24.8689643Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - -------------------------------------------------------
2020-07-19T09:16:25.7522091Z [ThreadedStreamConsumer] INFO org.apache.maven.plugin.surefire.SurefirePlugin - Running org.odpi.openmetadata.frameworks.connectors.ffdc.UserNotAuthorizedExceptionTest
2020-07-19T09:16:25.8914907Z Configuring TestNG with: TestNG60Configurator
2020-07-19T09:16:26.3467537Z [ThreadedStreamConsumer] INFO org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.581 s - in org.odpi.openmetadata.frameworks.connectors.ffdc.UserNotAuthorizedExceptionTest
2020-07-19T09:16:26.6907182Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - 
2020-07-19T09:16:26.6908480Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - Results:
2020-07-19T09:16:26.6909541Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - 
2020-07-19T09:16:26.6948277Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
2020-07-19T09:16:26.6949568Z [main] INFO org.apache.maven.plugin.surefire.SurefirePlugin - 
2020-07-19T09:16:26.8907276Z [main] INFO org.apache.maven.cli.event.ExecutionEventLogger -

Running with -X for debug doesn't provide any further hints beyond the execution of 3 tests As part of investigation into some other maven issues, I ran a partial build using gradle. Tests failed 

> Task :open-metadata-implementation:frameworks:open-connector-framework:test

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.TestAssetSchemaAttribute > testClone FAILED
    java.lang.AssertionError at TestAssetSchemaAttribute.java:139

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.TestAssetSchemaAttribute > testNullObject FAILED
    java.lang.AssertionError at TestAssetSchemaAttribute.java:162

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.TestAssetSchemaLink > testClone FAILED
    java.lang.NullPointerException at TestAssetSchemaLink.java:99

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.beans.TestSchemaAttribute > testClone FAILED
    java.lang.AssertionError at TestSchemaAttribute.java:92

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.beans.TestSchemaAttribute > testCloneSchemaElement FAILED
    java.lang.AssertionError at TestSchemaAttribute.java:92

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.beans.TestSchemaAttribute > testJSON FAILED
    java.lang.AssertionError at TestSchemaAttribute.java:220

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.beans.TestSchemaAttribute > testNullObject FAILED
    java.lang.AssertionError at TestSchemaAttribute.java:117

Gradle suite > Gradle test > org.odpi.openmetadata.frameworks.connectors.properties.beans.TestSchemaImplementationQuery > testJSON FAILED
    java.lang.AssertionError at TestSchemaImplementationQuery.java:129

705 tests completed, 8 failed

> Task :open-metadata-implementation:frameworks:open-connector-framework:test FAILED

Execution failed for task ':open-metadata-implementation:frameworks:open-connector-framework:test'.

The stats reported were:


705tests8failures0ignored0.854sduration | 705tests | 8failures | 0ignored | 0.854sduration | 98%successful
-- | -- | -- | -- | -- | --
705tests | 8failures | 0ignored | 0.854sduration

inspection shows we have many unit tests in the OCF -- far more than the 3 (!) that surefire runs in our current maven configuration

This needs investigation ... since it may be that we are not executing other tests that we should be....

planetf1 commented 4 years ago

Checked the sonar scans, which report on code coverage .. we can see the OCF tests coverage at https://sonarcloud.io/component_measures?id=odpi_egeria&metric=tests&selected=odpi_egeria%3Aopen-metadata-implementation%2Fframeworks%2Fopen-connector-framework%2Fsrc%2Ftest%2Fjava%2Forg%2Fodpi%2Fopenmetadata%2Fframeworks%2Fconnectors

This is consistent with the maven output -- only 3 tests are executed

Against 705 tests run by gradle ....

I then continued and got to repository-services-apis.

here maven does a lot better, reporting 224 successful, whilst gradle reports 271, 3 failing

AssetCatalog reports 0 tests run in maven, yet in gradle we execute 10, and fail 1 - again, Sonar code coverage confirms none of the code was tested at all

planetf1 commented 4 years ago

It's worth pointing out that the tests are executing with a default config so tests failing with gradle isn't unexpected or concerning in itself -- it was an experiment and there may be reasons why the environment is failing some tests especially where mockito and other mocking framework is used. However the fact the tests aren't even attempted in our usual build is the key take-away here that needs understanding

planetf1 commented 4 years ago

Archive of html reports from tests are at https://pastebin.com/0Z4Q0C0n

OUtput from ocf:

Screenshot 2020-07-20 at 18 21 21
planetf1 commented 4 years ago

Currently our maven surefire plugin is configured to run testcases:

<include>**/*Test.java</include>

https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html gives the defaults as

<includes>
    <include>**/Test*.java</include>
    <include>**/*Test.java</include>
    <include>**/*Tests.java</include>
    <include>**/*TestCase.java</include>
</includes>

Actually looking for file matches we only see at ffdc/UserNotAuthorizedExceptionTest.java a class which is actually a testNG test, has 3 testcases & does get run ok.

The other tests are in classes Test* -- so in this case our pattern matching is wrong - indeed removing the default override is probably the right step

When we do this we get expected output:

[ERROR] Failures:
[ERROR]   TestAssetSchemaAttribute.testClone:270->validateResultObject:139 expected [true] but found [false]
[ERROR]   TestAssetSchemaAttribute.testNullObject:182->validateNullObject:162 expected [true] but found [false]
[ERROR]   TestAssetSchemaLink.testClone:230->validateResultObject:99 NullPointer
[ERROR]   TestSchemaAttribute.testClone:180->validateResultObject:92 expected [true] but found [false]
[ERROR]   TestSchemaAttribute.testCloneSchemaElement:189->validateResultObject:92 expected [true] but found [false]
[ERROR]   TestSchemaAttribute.testJSON:220 Exception: expected [true] but found [false] expected [true] but found [false]
[ERROR]   TestSchemaAttribute.testNullObject:117 expected [true] but found [false]
[ERROR]   TestSchemaImplementationQuery.testJSON:129 Exception: expected [true] but found [false] expected [true] but found [false]
[INFO]
[ERROR] Tests run: 705, Failures: 8, Errors: 0, Skipped: 0

So this appears a simple mistake in the surefire plugin definition - easily fixed.

HOWEVER this cannot be merged as-is since unit tests are now failing. Further this will extend to other components too

Suggestions a) One person works on fixing all of these tests en-masse so the fixes and maven change can be checked in together b) Alongside a maven fix we configure maven to i) ignore all tests (too dangerous) ii) ignore selected module tests (hmmm) iii) ignore specific tests using the annotation @Test(enabled = false) c) fix all tests first, then merge the maven change

Although b) is the most changes, it's simple (done in 10s of minutes), and means from now on we will know if any tests fail. It also allows the test fixing to be incremental & distributed. This is my proposal

cc: @mandy-chessell

planetf1 commented 4 years ago

I've gone through the failing cases and addressed all with one of @Disabled (JUnit5) @Ignore (JUnit4) @Test(enabled=false) (TestNG)

I propose to create a PR with this change, and merge. The owners can then unpick the issue & we can leave this issue open to track - summary once build verified

planetf1 commented 4 years ago

Issues raised for components

Closing this as build process is now correct (subject to PR merge)