search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
806 stars 261 forks source link

Enable docker images to be built from local content (esp. for egeria container) #821

Closed planetf1 closed 5 years ago

planetf1 commented 5 years ago

Enable a developer working in their dev environment (macos, linux, windows) to test changes in a cloud based k8s service

For example

scenarios I've felt the need for this in

For example

See also #810 which allows for local containers to directly use locally built artifacts

planetf1 commented 5 years ago

Will look at this shortly as the slow turnaround time was an impact at a recent workshop testing integration with egeria. The approach I will investigate first will be to use maven assets as the source of the jars added to the image. This keeps the docker build somewhat decoupled yet allows for rapid turnaround either from local m2, odpi repo, azure pipeline repo or public maven central

planetf1 commented 5 years ago

I have a reworked egeria dockerfile which builds from the latest maven repo version

By default this uses 1.1-SNAPSHOT & the nexus public repo (where our build snapshots go) but these are arguments which can be modified by the user.

This cuts the docker build time massively - to just over 30s for the build phase which was taking around 30 minutes. More importantly it means that a) The artifacts used in the container environment are identical to that we build on the official build b) We test the distribution archives that are intended for other consumers too.

For now I am using the open-metadata-assemblies maven artifact. This contains ALL of our .gz archives. The only one containing the user interface is the *distribution.tar.gz

This works but future changes might include

So will leave this issue open for that consideration.

Now testing the new build approach....

planetf1 commented 5 years ago

The docker build fails following this change....

The reason is that the order the plugins executed in the pom is

This is a reasonable default as usually one would be building java code into a jar, then creating an external zip archive for download

However in our case we are trying the opposite - to take our assemblies, and package them into a special packaging jar for distribution via maven central.

In order to achieve this I have re-configured the maven assembly plugin to execute in the 'prepare-package' phase rather than 'package. Since the jar build is linked to the 'package' lifecycle phase this will correct the order. I have then added the 'default-jar' build of the jar which will take the archives and compose them into the module jar.

It likely worked locally due to not working in a from-0 build environment and testing steps in a unit test manner.

planetf1 commented 5 years ago

So we now have

➜  open-metadata-assemblies git:(dockbldfix2) ✗ jar -tf target/open-metadata-assemblies-1.1-SNAPSHOT.jar
META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.odpi.egeria/
META-INF/maven/org.odpi.egeria/open-metadata-assemblies/
egeria-1.1-SNAPSHOT-distribution.tar.gz
egeria-1.1-SNAPSHOT-sources.tar.gz
egeria-1.1-SNAPSHOT-omag-server.tar.gz
META-INF/maven/org.odpi.egeria/open-metadata-assemblies/pom.xml
META-INF/maven/org.odpi.egeria/open-metadata-assemblies/pom.properties

Note: as we revise our packaging this may be revisited as per earlier comments, but it moves the docker build forward so is a prudent change for now.

cmgrote commented 5 years ago

Is the docker image build expected to be fixed now? Docker Hub image hasn't been updated in ~3 days -- so it's missing various merged PRs now...

12:59:36 [INFO] gzip: egeria-1.1-SNAPSHOT-distribution.tar.gz: No such file or directory
12:59:36 [INFO] 
12:59:36 [INFO] tar: This does not look like a tar archive
12:59:36 [INFO] 
12:59:36 [INFO] tar: egeria-omag-1.1-SNAPSHOT/user-interface/access-services-user-interface-1.1-SNAPSHOT.jar: Not found in archive
12:59:36 [INFO] tar: egeria-omag-1.1-SNAPSHOT/server/server-chassis-spring-1.1-SNAPSHOT.jar: Not found in archive
12:59:36 [INFO] tar: Exiting with failure status due to previous errors
12:59:36 [INFO] 
12:59:36 [ERROR] The command '/bin/sh -c mvn org.apache.maven.plugins:maven-dependency-plugin:2.5.1:get -DremoteRepositories=$repo     -Dartifact=org.odpi.egeria:open-metadata-assemblies:${version}:jar -Ddest=egeria.jar &&     jar -xf egeria.jar egeria-${version}-distribution.tar.gz &&     gunzip -c egeria-${version}-distribution.tar.gz | tar -xf -        egeria-omag-${version}/user-interface/access-services-user-interface-${version}.jar       egeria-omag-${version}/server/server-chassis-spring-${version}.jar &&     mv egeria-omag-${version}/user-interface/access-services-user-interface-${version}.jar /opt/egeria/binaries/ &&     mv egeria-omag-${version}/server/server-chassis-spring-${version}.jar /opt/egeria/binaries/' returned a non-zero code: 2
12:59:36 [INFO] ------------------------------------------------------------------------
12:59:36 [INFO] BUILD FAILURE
12:59:36 [INFO] ------------------------------------------------------------------------
12:59:36 [INFO] Total time: 01:24 min
12:59:36 [INFO] Finished at: 2019-09-19T11:59:36Z
12:59:36 [INFO] ------------------------------------------------------------------------
12:59:36 [ERROR] Failed to execute goal com.spotify:dockerfile-maven-plugin:1.4.10:build (build-and-tag-latest) on project open-metadata-docker-egeria: Could not build image: The command '/bin/sh -c mvn org.apache.maven.plugins:maven-dependency-plugin:2.5.1:get -DremoteRepositories=$repo     -Dartifact=org.odpi.egeria:open-metadata-assemblies:${version}:jar -Ddest=egeria.jar &&     jar -xf egeria.jar egeria-${version}-distribution.tar.gz &&     gunzip -c egeria-${version}-distribution.tar.gz | tar -xf -        egeria-omag-${version}/user-interface/access-services-user-interface-${version}.jar       egeria-omag-${version}/server/server-chassis-spring-${version}.jar &&     mv egeria-omag-${version}/user-interface/access-services-user-interface-${version}.jar /opt/egeria/binaries/ &&     mv egeria-omag-${version}/server/server-chassis-spring-${version}.jar /opt/egeria/binaries/' returned a non-zero code: 2 -> [Help 1]
planetf1 commented 5 years ago

I am aware of the issue. I've manually pushed an update to dockerhub with latest master. Will fix soon.

planetf1 commented 5 years ago

I can confirm the build is now fixed - it's building under jenkins (and also azure) and down to around 3-5 mins by reusing maven artifacts. I am considering whether we can trigger the container build automatically after, or together with a merge build (not verify which is on a PR). Ideally an optional PR trigger would work too - but publish to a diff repo. Will raise an issue on some of these ideas

planetf1 commented 5 years ago

See https://dev.azure.com/odpi/Egeria/_build/results?buildId=199 for the azure builds. I'm inclined to close this issue and open new ones as the changes went slightly away from the original text in the issue (build from local) to improving the build performance/turnaround time to integrate fixes.