Egeria docker image is incomplete (samples, utilities) - missing full distribution archive

The docker image should contain the full distribution archive, yet:

jonesn:egeria/ (master) $ docker run -it odpi/egeria:latest /bin/ls -la /opt/egeria                                [9:49:22]
total 156
drwxr-xr-x    1 egeria   egeria        4096 Jun 17 21:38 .
drwxr-xr-x    1 root     root          4096 Jun 17 21:38 ..
-rw-r--r--    1 egeria   egeria       11357 Jun 17 21:37 LICENSE
-rw-r--r--    1 egeria   egeria         268 Jun 17 21:37 NOTICE
drwxr-xr-x    2 egeria   egeria        4096 Jun 17 21:36 clients
drwxr-xr-x    2 egeria   egeria        4096 Jun 17 21:37 conformance-suite
drwxr-xr-x    4 egeria   egeria        4096 Jun 17 21:36 server
-rw-r--r--    1 egeria   egeria      113793 Jun 17 21:37 truststore.p12
drwxr-xr-x    3 egeria   egeria        4096 Jun 17 21:37 user-interface
drwxr-xr-x    2 egeria   egeria        4096 Jun 17 21:36 utilities
jonesn:egeria/ (master) $ ls -la ~/src/egeria/open-metadata-distribution/open-metadata-assemblies/target/egeria-2.0-SNAPSHOT-distribution/egeria-omag-2.0-SNAPSHOT
total 256
drwxr-xr-x  11 jonesn  staff     352 19 Jun 17:32 .
drwxr-xr-x   3 jonesn  staff      96 19 Jun 17:32 ..
-rwxr-xr-x   1 jonesn  staff   11357  5 Nov  2019 LICENSE
-rwxr-xr-x   1 jonesn  staff     268  5 Nov  2019 NOTICE
drwxr-xr-x  44 jonesn  staff    1408 19 Jun 17:32 clients
drwxr-xr-x   5 jonesn  staff     160 19 Jun 17:32 conformance-suite
drwxr-xr-x  18 jonesn  staff     576 19 Jun 17:32 samples
drwxr-xr-x   5 jonesn  staff     160 19 Jun 17:32 server
-rwxr-xr-x   1 jonesn  staff  113793 17 Jun 16:09 truststore.p12
drwxr-xr-x   4 jonesn  staff     128 19 Jun 17:32 user-interface
drwxr-xr-x   8 jonesn  staff     256 19 Jun 17:32 utilities
jonesn:egeria/ (master) $ ls -la ~/src/egeria/open-metadata-distribution/open-metadata-assemblies/target/egeria-2.0-SNAPSHOT-distribution/egeria-omag-2.0-SNAPSHOT/utilities
total 122808
drwxr-xr-x   8 jonesn  staff       256 19 Jun 17:32 .
drwxr-xr-x  11 jonesn  staff       352 19 Jun 17:32 ..
-rwxr-xr-x   1 jonesn  staff  20271631 18 Jun 09:53 glossary-canonical-model-2.0-SNAPSHOT-jar-with-dependencies.jar
-rwxr-xr-x   1 jonesn  staff     21772 18 Jun 09:53 glossary-canonical-model-2.0-SNAPSHOT.jar
-rwxr-xr-x   1 jonesn  staff  35025924 18 Jun 09:53 open-connector-archives-2.0-SNAPSHOT-jar-with-dependencies.jar
-rwxr-xr-x   1 jonesn  staff     10840 18 Jun 09:53 open-connector-archives-2.0-SNAPSHOT.jar
-rwxr-xr-x   1 jonesn  staff   7336559 19 Jun 17:28 open-metadata-types-2.0-SNAPSHOT-jar-with-dependencies.jar
-rwxr-xr-x   1 jonesn  staff    193350 19 Jun 17:28 open-metadata-types-2.0-SNAPSHOT.jar
jonesn:egeria/ (master) $ docker run -it odpi/egeria:latest /bin/ls -la /opt/egeria/utilities                      [9:50:19]
total 7364
drwxr-xr-x    2 egeria   egeria        4096 Jun 17 21:36 .
drwxr-xr-x    1 egeria   egeria        4096 Jun 17 21:38 ..
-rw-r--r--    1 egeria   egeria     7333065 Jun 17 21:36 open-metadata-types-2.0-SNAPSHOT-jar-with-dependencies.jar
-rw-r--r--    1 egeria   egeria      195525 Jun 17 21:36 open-metadata-types-2.0-SNAPSHOT.jar
jonesn:egeria/ (master) $

docker inspect of the image shows a few entries like:

       "Id": "sha256:d27930bd5857b51f8f8cfb0c8f7af06db64811613e324510d880b85f9d2e5d4e",
        "RepoTags": [
            "odpi/egeria:latest"
        ],
        "RepoDigests": [
            "odpi/egeria@sha256:ae3083bdb9efd902b39eecac5eea84574ba1581e99bd1d90a9eda79fe423be7b"
        ],

        "Created": "2020-06-17T21:38:33.239829902Z",

            "Image": "sha256:5937985f4105c6b9d7164a2db5a357bdffe98d95482bf2d24944da7ae97386e4",

org.opencontainers.image.revision = e764e5d       org.opencontainers.image.licenses = Apache-2.0       org.opencontainers.image.created = 2020-06-17T21:36:56+0000       org.opencontainers.image.version = 2.0-SNAPSHOT       org.opencontainers.image.documentation = https://egeria.odpi.org/open-metadata-resources/open-metadata-deployment/docker/egeria/       org.opencontainers.image.ext.vcs-date = 2020-06-17T21:13:17+0000

These show the images are current

The impact to this will be that samples, utilities are missing, impacting the ability to extend the current tutorials including fixing labs like odpi/egeria-jupyter-notebooks#34 - however it is not affecting the remainder of the labs at this point

The issue is simply that the assemblies are not marked as dependent on some modules including where the samples, cim import etc are. This means the assembly gets built BEFORE these extra samples/utils - so they are not included ON THE FIRST RUN. On subsequent runs - ie in the developer's environment, they will be picked up on asecond run - albeit an old version ....

Of course the official build is clean every run, so the extra will always be missing

The fix should be to add the correct dependencies.

This brings up an interesting issue -- how exactly to express the right dependencies to ensure everything needed is built ready for assembly, and that other modules are then built using this assembly.

Most of our code is in open-metadata-implementation. However we also have some components under open-metadata resources, such open-metadata-archives (utilities for CIM information model etc), lab notebooks in open-metadata-labs & a series of samples in open-metadata-samples including for example the coco pharma security provider.

With the main assembly built, our docker image inserts this into the image. The docker image themselves are then used by our helm charts and docker-compose .. yet these modules themselves are under open-metadata-resources/open-metadata-deployment

Currently the assembly is not really dependent on anything, but happens to work. Simple adding dependencies on the obvious 'parent' pom is actually not sufficient - for example see https://stackoverflow.com/questions/11131575/build-order-of-maven-multimodule-project. The files were include are referenced by relative path to the assemblies directory (defeating the dependency mechanism)

One approach is to ensure that WHENEVER we include a file into the assembly, we must include the exact artifact that originates in -- not a parent , but the actual artifact as a dependency. It appears though the assembly descriptor already effectively supports this - see https://maven.apache.org/plugins/maven-assembly-plugin/examples/multimodule/module-binary-inclusion-simple.html - we should be replacing file paths with module dependencies.

A slightly cleaner way may be to assume most things we build could contribute, and restructure so that a parent project does the assembly (and makes use of useAllReactorProjects as per https://maven.apache.org/plugins/maven-assembly-plugin/examples/multimodule/module-binary-inclusion-simple.html), with all our implementation (+samples etc etc) as submodules. Docker images/charts would be out of that tree, but this is a significant rejigging of the directory structure. Even this has some challenges as per https://maven.apache.org/plugins/maven-assembly-plugin/faq.html#module-binaries

I will look at refactoring the assembly to

detect errors if possible
define moduleSets per contributing dependency

I hope this will be the 'correct' way to let maven properly resolve dependencies and include just what is needed (without getting into circles unnecessarily). An additional thing to check is that our FVTs (which depend on open-metadata-assemblies) continue to be launched at the correct time, since they are dependent on the assembly, just as our docker image is.

Unfortunately our structure is now complex enough that the initial incorrect useage is now breaking

Note - have split out corrections/improvements to the assembly we use for jupyter (notebooks) and the node based ui/presentation server to seperate issues as there are other factors to look at.

The changes for this issue will focus on the core egeria distribution.

An update on where this work is at so far

The overall principle has been for assemblies to use maven artifacts correctly, rather than relative paths. All paths are then relative to each module so survive any refactoring better, plus the specification is much clearer in which components are being pulled into an assembly
The resulting assembly is simpler and clearer, with sections for server, samples, clients, utilities, cts etc
The assembly now has correct pom dependencies so we ensure modules are built before we try to include them - and before there was no error checking/warning - hence this original issue.
Within the dependency, server/lib, as an example, includes not just the jars for any connectors we build, but all of THEIR dependencies, which are needed to actually allow the connectors to run. Similarly, clients will also contain not just the -client jar, but also all of their dependencies, of which there are a lot
We can now make use of more assembly capability such as automatic dependency inclusion

However there are a few non trivial issues

The build of the assembly is now taking 10 minutes instead of 1. I am looking at persuing this with the maven team, but it looks like the huge dependency chains we are dealing with and the large number of modules (>300) are causing high processing times.Debug shows we are not hitting a stall, it's just the plugin is doing an awful lot of work
60% of the time is building the .gz which is needed to add an additional attachment to an artifact -- and this is used downstream by the docker container and fvt tests
- Looking at the current distribution we have a 1.6GB .tar.gz
- The unpacked tree is shown at https://gist.github.com/6db844756f3fe6e2ac29a8e0538dc57b
- Note also there is a bug report https://issues.apache.org/jira/browse/MASSEMBLY-938 - currently the impact is only a warning however
- Furthermore the investigation has uncovered more issues in the prototype maven build for the node-based UI - fixing that is hived off to a new issue as it is more a design discussion about what it means to distribute a UI.

How we correct this? I think we need to look again at project dependencies. A few ways:

at the assembly level we can use includes & excludes to fine tune - though this is quite error prone. We could iterate through and pick off any we don't think are needed
We could skip all automatic inclusion of dependencies and manage manually - extremely error prone
we can look at the modules themselves, and try and unpick any undesired dependency chains that are unnecessary/incorrect (mvn dependency:tree, and carefully examine each....)
We can try and build rules, heuristics, and try and build a special plugin which can be more rule driven (probably a lot of work)
We can move to dynamic loading of connectors, which will also reduce the dependency chains, but note that to work, that change does depend on this one.
We could further reduce dependencies by declaring some dependencies as 'provided' for connectors. this means we tell maven 'trust us - they'll be there', so this would mean everything in the server chassis for example, which is currently packaged by spring in a single jar would not be duplicated in server/lib as an unpacked file (mostly egeria* components)
For clients/samples/utilities we could continue with jar-with-dependencies in the distribution and take the hit on duplication. We could even do the same with connectors, but the result will almost certainly be an explosion in size to way above that of the current assembly archiv
We could abandon much of this change and stick with specifying files directly. This avoids the build time hit. We can still fix build ordering mostly by adding additional dependencies. But we'll have less error checking, and still need to fix dependency management for loadable connectors.

At this point the server is seeming to be working correctly.

In essence I think this is doing things closer to 'the right way' and is actually exposing hidden issues we currently have

At this point I'm looking for feedback on the general direction. We need to consider whether these changes make sense as a direction. Then, if so, whether we can integrate now & work on mitigations later or if we need to hold back until more of these issues are addressed

See https://github.com/odpi/egeria/pull/3316 for the current PR that implements these changes (just finalizing a few build errors)

On investigation it appears we are getting every dependency pulled in regardless of what modules are put in any directory of the distribution. Following up with maven team -> https://lists.apache.org/thread.html/r131e0699cb3d60763d0904425491924097168bb2de68da24bb5ddcb5%40%3Cusers.maven.apache.org%3E

Opened up another issue with maven: https://issues.apache.org/jira/browse/MASSEMBLY-940 - the dependencies being loaded are wrong. They should be base off a moduleSet (as best I can determine) NOT project.

Looking for alternatives.

I've looked again at how we might improve the maven based assembly particularly to any shorter term solutions

Without getting deep into maven and fixing/forking the assembly plugin or building a custom plugin, the assembly build is limited.

We can only pull in all assemblies from the POM level (of the assembly) or none. We cannot do moduleset specific including (ie dependencies of connectors in once place, clients in another, utilities in a third etc)

The maven download, dependency, build-helper plugins have some overlapping capabilities, but only work on a single module that is being built, there's no real way of doing aggregate changes/merges across a set of modules.

The 'provided' scope can help when building a single module as we can say 'that's ok - it's already in the platform', but since our modules can get used in different ways and pulled in transitively it can be tricky to specify this correctly which is particularly tricky without extensive testing/validation of different components at least in terms of assembly structure & dependencies.

Ideally for example under server/lib we might want

to add our connectors
to include their dependencies
but skip any already built into the server chassis (since this is where they run)

Clients on the other hand need to be standalone to we probably want

all clients
with all dependencies

We might conceive of other structures or common dirs, & building aggregate jars, but this flexibility doesn't seem possible.

There seem to be few options that can be done simply as a near term measure a) Always create & use 'uber jars' for anything pluggable such as connectors, being used in an unknown way (clients), standalone (utilities etc). This probably has the least impact. We already build these jars, we just need to include them in artifacts, and then the assembly. Very little work and reliable at the expense of size/time

b) Have a general 'lib' directory into which we add ALL dependencies - but this will be > 400 including duplication of server components. Again only minor change to the build/server to ensuring the correct LOADER_PATH is specified, but it would mean client/utility uses specifying classpath correctly or still using some uber jars. Overall not as simple. There is also a noteable additional impact on build time (5-10 minutes) in evaluating all the dependencies together due to their complexity

I plan to close this out persuing option a) if no comments follow so we can fix the initial issue

A possible medium term measure is to revisit packaging & dependency management across the product so that we focus more on standard 'packages' (we have some already, but don't use them, and they may be incomplete), and have a very clear perspective on which modules make up different 'platforms'. We then shift our dependencies away from fine-grained to this coarser grained level. We can then more easily make use of 'provided' scope for either common bundles or the server platform, and reduce the list of extra dependencies or increasing in size of uber jars. We might wish to do this alongside reviewing our overall build/packaging approach as it's likely to need more investment in experimentation and flexibility in our build. (see also odpi/egeria#3371). (Note also that any changes here currently do quite increase build time.)

Or we may break the direct 1:1 mapping between the jars we build and our maven artifacts outside maven itself so that the 320+ do not get published - we won't have them. Instead we keep the 320+ source jars and assemble them on-demand as part of a build process into the more select 'package' type bundles that make sense. This also could address other process challenges but would depend on a radical change like odpi/egeria#3371

So really More medium term more review of this area is needed & elaboration of the use caseswith a broader look across all artifacts (ie including docs) ie

Java developer building extensions to egeria or including in their tech - mostly maven driven
Someone deploying egeria - focussed on server/connectors & how to add additional ones
etc etc

I think that is best served by a fresh issue & analysis & being less concerned on any current technical constraints of the build process & more concerned on the end users. We can fix the near term issue here.

As per odpi/egeria#3458 we will

drop clients
place all dependent jars into a common lib directory
include samples/utilities as uber jars

This has now been implemented. closing

odpi / egeria

Egeria docker image is incomplete (samples, utilities) - missing full distribution archive #3231