maven directory changes

[grantmacken]

New maven based layout for exist-db, will require an update of the copy operations and perhaps the sym linking of the config files, in the Dockerfile.

I have opened this issue to check list and discuss what is required

[duncdrum]

with the switch to maven we need to first decide if maintaining a separate repo for the docker containers still makes sense. I m leaning towards integrating the docker operations with the build process in the main repo via a maven plugin.

Move …

• Going with spotify we can build and push in e.g. travis, using encrypted env variables for the credentials necessary for publishing new images to the container registry. This way we don't need every core-developer to register on dockerhub.
• Going with fabric8io we d be able to run the containers as part of the build for integration testing, which sounds appealing to me.

In both cases, I would want to stick with dockerfile based builds, no funky syntax pls.

Modifying conf files could happen via standard maven variable substitution which would reduce the number of layers for our images, but is something we need to prepare.

This involves sunsetting this repo, changing the configuration of our build pipeline on dockerhub, and modifying files in the main repo.

pros:

• tighter integration with regular exist development
• less code maintenance
• not sure what happens if ppl clone the repo, and run maven without having docker installed, or how to this setup will handle docker credentials for those that have them.
• integration tests can be run locally, lots of potential imv

cons:

• more work
• porting our tests could be tricky
• tighter integration = less separation of concerns
• int testing sounds promising but might turn out to be much ado about nothing

Don't move …

• need to switch to builder image that has maven
• need to completely rewrite the dockerfile
• need to adjust the paths to conf files etc.
• need to reassess the minimal build target, and see where things might need adjustment in the main pom.xml
• conf file modification could still be handed over to maven, no more xmlstarlet in builder, and faster builds.

We should also monitor the image size, i see that our images are consistently increasing in size, and i don't see a good reason for why they should. If one of the two approaches leads to substantially smaller images, that one takes the cake for me.

Lastly, both the docker build commands, and the dockerhub code have undergone major updates recently, I m still adopting to the fallout of that, there are likely unforeseen bumps ahead.

[adamretter]

not sure what happens if ppl clone the repo, and run maven without having docker installed, or how to this setup will handle docker credentials for those that have them.

Well you can either not add the docker-maven module to the reactor build (i.e. the root pom.xml), or you could use a Maven profile for the docker stuff which is not enabled by default. The profile can be enabled automatically by detecting Docker somehow or by the user setting a property when invoking maven (see https://maven.apache.org/guides/introduction/introduction-to-profiles.html).

We already use a profile for code-signing the DMG on a Mac. That profile is only enabled when both the OS is a Mac, and the user has set the property mac.signing=true e.g. mvn -Dmac.signing=true clean package

[grantmacken]

I am for moving to main repo and the docker image being a release artifact of the maven build process. If we are going to use a maven plugin to simplify this process then I prefer https://github.com/spotify/dockerfile-maven

@duncdrum

Don't move …

need to completely rewrite the dockerfile

Whatever we do, a rewrite of the Dockerfile is required.

not sure what happens if ppl clone the repo

I don't think, we should have any focus on this. What we want to produce is docker images as release artifacts, along side other release assets like the install binary , .dmg or homebrew package.

If people want to create their own eXist image then our image should be the preferred base image they start FROM in their own Dockerfile. If people don't want a distroless container, then I expect people can start from the install binary to build a eXist image, like I do with the alpine image.

[duncdrum]

ok so move with plugin it is for now. I m currently working on two branches to set this up (one with fabric8 plugin one with spotify).

I should have something ready for testing tonight, but here is my current list of criteria:

functionally use as base image is crucial, this involves:

• sensible upgrade path for existing images,
• functional equivalence with respect to FOP
• logging via docker logs
• user modifiable conf files, i.e. you shouldn't have to clone and build the main repo to get a docker image with a new conf.xml
• proper tagging of releases
• image size should stay comparable to what we have now
• maintain specialized jvm config for container applications (this is trickier than it sounds, since the jvm is now part of maven build config and not simply what on a machine, and there are syntax changes depending on the point releases of java8, let alone 8 vs 11)
• images need to be tested before being pushed to the registry

the biggest changes so far:

• client.jar is no more
• appassemblerBuilder is quickly shaping to be the next bane of my existence
• autodeploy path changes

[grantmacken]

maintain specialized jvm config for container applications

As I understand it these jvm and java tool options are for adjusting for differing server environments re. memory allocation etc. They can always be set at docker run time, however there are probably sane defaults which you are aiming for. Perhaps in the maven test env, there could be some under load stress test to get a baseline memory allocation profile, so build comparisons can be made.

[duncdrum]

not quite the server environment doesn't matter, they are for making sure that the internal JVM memory mechanism and the external container memory mechanism don't get into conflicts, how a server allocates memory to the container daemon is yet another level on top of that.

with java11 the JVM should automatically discover when it is running inside a container environment, and apply the righ settings, but with java8 we need to configure these via JVM_OPTS. Setting at docker run time only works if the image makes the appropriate provisions

[duncdrum]

here is what i have so far with farbic8io the classpaths still need to go the image is still in prototype stage, default apps are MIA, no clue why and if this is due to docker or a general problem of the new build working on conf modifications and understanding the flow from exist-distribution/src/main/conf/ via appassemblerBuilder to exist/etc