Automate the FVT tests

[davidradl]

use a container approach similar to what is proposed for the proposed Conformance test automation

[planetf1]

We should look at automating the deployment of a k8s environment based on egeria that we can run automated tests against, configured from within the build pipeline

First step:

• Identify/fund a k8s environment/cluster

[planetf1]

Results of fvt test runs should be stored as some kind of build output artifact so that it can be referred to both by developers (to see how the last test run went) and for consuming organisations wanting to see the results from the current release.

[planetf1]

Looking at what we have under FVT currently:

• open types gets built and run & is now quick and reliable
• subject area fvt can be run ( see also #2693) in part -- the single server solution, The multi server test requires infra (see #747) but this could be satisfied by the typical coco environment
• asset consumer fvt could be targetted at a single server

One we have a deployed environment and pipeline we can return and link these with the runtime

[planetf1]

In #3245 we had some confusion over which tests to run, and in #3254 around the environment.

It's excellent we have some FVT tests, it's clear a little time invested in running these tests automatically (ie via surefire) with the likely needed refactoring will pay off, and we can extend the model to other tests.

I think this will supercede #747 so will close that

Aiming to 2.1 timeframe.

[planetf1]

Our current unit tests use the 'surefire' plugin. If a test failure occurs the build immediately aborts. Whilst the surefire plugin could be used to execute any kind of test, it's not well suited to having pre and post environment setup steps prior to the actual test. For example launching the egeria chassis or tearing it down.

Failsafe (an anagram of surefire) allows this, and the maven faq positions failsafe as more appropriate for integration testing. https://maven.apache.org/surefire/maven-failsafe-plugin/ https://www.baeldung.com/maven-failsafe-plugin https://stackoverflow.com/questions/28986005/what-is-the-difference-between-the-maven-surefire-and-maven-failsafe-plugins

It can still run junit, testng, pojo etc tests , and just as with surefire, coverage data can be captured to be used with Jacoco & potentially onwards to sonar.

Therefore I am proposing we gear functional tests around failsafe. These will be tests that require some running components to execute - so are more complex and may take longer than unit tests.

Tests will not be run by default (mvn clean install) but will be run with 'mvn verify' . This can be added to the merge build, but most likely not the PR build due to time taken (we can check and add after if wanted).

failsafe's convention is that tests are named: "/IT*.java" - includes all of its subdirectories and all Java filenames that start with "IT". "*/IT.java" - includes all of its subdirectories and all Java filenames that end with "IT". "/*ITCase.java"

Additionally the plugin looks for these files under src/test/java

These can be overridden - see http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html , though it would be preferable to keep to these convensions -- especially for new tests -- to aid code clarity by adhering to convention.

This brings us to the current Subject Area FVTs and how they might be integrated. Currently they are in a standalone project under open-metadata-test/open-metadata-fvt

Egeria is a multi-module maven project. When we run 'mvn install' for example, maven figures out a dependency order, and then executes to the install lifecycle phase in each module in turn. If we go for 'mvn verify' then it will do the same up to the verify lifecycle stage. So it does not do every project's 'clean' then 'prepare' then 'compile' etc.

This means we need to be careful with where Integration tests go. If they are truly dependent on the entire chassis we must ensure we depend on that component. Or indeed if we need to use the assemblies to run egeria we must depend on that (as long as the assemblies don't need to then pick up test results or similar... causing a circular dependency ....). Starting with a clean high level tree which is easier to make dependent on everything else (like the assembly) seems like the way to start. Of course this gives us a maven module with no source - something maven central will object to....

Also Each suite of tests (in a directory) is independent -- unfortunately this means any pre/post steps - which could include starting a server are done for each suite. This could be slow.

It seems as if maven isn't ideally suited and we are pushing the boundaries...

My proposal is therefore

• Use the failsafe plugin
• Focus on the subject area tests, ensuring the server is started before/after [skip the two server test]
• Modify the failsafe plugin to use the existing filenames if possible (ie minimal changes)
• Add any minimal parameterization needed
• -> review

I'll work on the basis of creating a PR, we can discuss during the process/at review status if we need to make a change of direction

Subsequent work could

• refactor/rename/move the tests to adopt standard naming guidelines
• Optimize any start/stop time by seeing if we could move that to high level modules
• • or look at why egeria startup time is significant

There are also more complex test automation needed around deploying to k8s, testing a full coco pharma environment, perhaps browser testing with selenium or similar -- and this likely goes way outside maven's sweet spot, so this is just a first step to improve our test coverage/execution Any comments? cc: @davidradl

[planetf1]

A little more evaluation of various options & some experimentation of a few techniques:

• Able to configure failsafe in pom without issue
• The pre/post stages for a test can be bound to pre-integration-test and post-integration-test phases - main gotcha is ensuring any new processes are cleaned up if the test is killed before it shuts down the server
• Tried groovy maven plugin for java like scripting - able to launch an egeria instance, check for life (server origin) - but a) awkward to use extra modules b) maven plugin isn't very flexible - ie working directory c) little more concise than java d) language dated e) adds another plugin dependency- though we already use groovy at runtime, indirectly
  • Launching egeria is complicated by the new security configuration. Specifically if our truststores are not in the current working directory (the groovy plugin wouldn't directly allow cwd to be changed) the .p12 files are not found. Both the spring setting (for the chassis) AND the java level setting is needed to work correctly (elastic search fails otherwise). (this is before the second phase of security work)
  • There's a dedicated springboot plugin which can start the chassis -- but a) the environment will differ from our assemblies - which seems less desirable b) Would really need to be part of the server-chassis-spring module
• Also briefly looked at cucumber (framework), jython for scripting (language is dead)/ python (dependencies), java shell, other maven plugins that can launch executables/run REST api
• Running via docker/k8s is simpler & provides good management of the platform (ie easy to create deployments, charts, check for life, and when we add an operator, easier to create a server - hopefully!)- IF you have the infra - but is likely an inhibitor for many developers to run tests frequently.
• conclusion - simple approach is a) use antrun to start/stop server - though ant is dated, it's best integrated with maven, already used, has flexibility over the environment like working directory, and our usage will be simple b) add some java utility classes/modules to configure server (similar to admin client sample, a precursor to a CLI) - using the egeria admin client recently written. Additional classes will need to be in a new module, maybe multiple, so we can correctly add dependencies for running the test. This is still worth doing even if subsequently decide to move to a different framework or focus more on container based testing by keeping simple testing simple....

[planetf1]

A quick update on status

• I can reliably (first-pass!) start/stop/clean an egeria platform & server for executing tests against.
• Configuration of the platform/server is done via a groovy script
• In part, Due to #3291 the current config is REST based, - moving to client can be done later
• The tests currently are POJO based with run() methods optionally taking parms.
• When running with a plugin we need to inject any config via system properties - so as a poc, an additional method can be added with hardcoded values - can modify to use system properties later
• The main problem at the moment is persuade surefire to exit the correct methods in the right classes (as well as the right tree - though this works ie we use src/main/java not src/test/java) . This should work, but after hitting dead ends, I've now contacted the maven team for help (https://lists.apache.org/thread.html/r00d0a6a3fe39327ecf494136d09a3cef0d1e20387db4e0a24ff9f280%40%3Cusers.maven.apache.org%3E ). I'm still hoping this will be successful
• Most failsafe users seem to be using JUnit or testNG and this is probably a better model for our tests, including integration, moving forward.
• Sticking with convention/defaults in naming - main vs test, class/method names will make things much clearer. I propose we try this on the next set of tests, and then can come back and refactor these.
• The SA fvt tests utilise 2 servers for some tests. I'm not working with these currently. This makes for a much more complex environment with kafka, zookeeper as well as multiple servers. technically it's possible, but it may be more appropriate to run those tests in k8s or similar as another level of system integration testing.

[planetf1]

I've now created a PR #3297 with the first running version. The PR has extensive notes on what it does & limitations.

In terms of general discussion ... I was not able to get failsafe to run POJO tests reliably. Many approaches were tried. Specifically the problem was in failsafe identifying which classes and methods to use. In theory this should work, and in discussion on the mailing lists I tried a test project which worked ok, but I was unable to translate this into egeria. Even then I'd need to create a new POJO method to run the test easily without parms (but with the parms I needed)

The 'natural' approach for failsafe is to use a test framework like JUnit5, and to have tests defined in 'src/test/' By switching to this model tests were found reliably . The current FVTs were not written as JUnits, so the first pass was to add a wrapper test method to demonstrate the approach - though as the PR notes, more work is needed on at a minimum ensuring errors are correctly returned.

For further tests I think the idea of having an integration test directory distinct from the rest of the code is sound, but we should use the test directories, and write as JUnit(5)s. I suggest we take a another batch of tests to apply that approach to, and in the process can consider adding docs & refactoring poms etc to reduce duplication when adding new FVTs.

[planetf1]

The first step of this has been done.

Any specific changes for subject area can be covered in a subject area specific issue

Next steps

• Identify another area to validate the approach (cc: @mandy-chessell )
• refactor (poms, scripts if needed) as needed to support more general use across components
• Use admin client to configure server under test
• Consider enforcing checks in the build (PR: time dependent)
• Developer Guidelines
• Publishing test results
• Capturing code coverage and augmenting existing coverage data

[planetf1]

FVTs running in merge build perfectly, so the approach works :-)

only a few seconds currently:

[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - Subject Area OMAS FVT .............................. SUCCESS [ 36.095 s]

see https://dev.azure.com/ODPi/Egeria/_build/results?buildId=8410&view=logs&j=ac0a620f-bc12-5590-dc41-1227e7934d92&t=20d5b08f-cc4c-56d9-a78d-7c0974959311

[planetf1]

Another observation - we can increase startup time of the egeria chassis further by ensuring we set springdoc.api-docs.enabled=false (or export SPRINGDOC_API_DOCS_ENABLED=false) Will look at adding to existing invocations on next round of changes (cc: @davidradl )

[planetf1]

Next action on this - add parm above & add docs

[planetf1]

this is done