Ranger Docker Build Fails Due to Missing Dependency

[RyanHolstien]

Failure to find org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde

The above dependency is resulting in the Apache Ranger Docker build to fail as this artifact is not present in https://odpi.jfrog.io/odpi/egeria-snapshot/org/ which is set as the repository to pull from in egeria/open-metadata-resources/open-metadata-deployment/docker/build-images/apache/ranger/dist/settings.xml .

I was able to get around this for myself by adding a local repository to the list of repositories in the settings.xml hosting the artifact. Earlier today I saw an empty stub for org/pentaho in artifactory, but it is no longer there so something a bit strange appears to be going on there.

To fix properly the artifact needs to be re-added to artifactory, I do not have permissions to do this so it will need to be fixed by a project maintainer.

[planetf1]

Thanks for reporting.

Our automated builds for the docker images are at : https://dev.azure.com/odpi/Egeria/_build?definitionId=13&_a=summary - these are still ok.

• Only master is built currently
  • Whilst we deploy ranger in the 'vdc' helm chart, it's not configured to talk to the other components out of the box - that needs setting manually
  • There are two ranger builds - first the base, then another build (with our configuration)
  • The -build images are only built manually as they contain only a fixed release of code (like ranger 1.2)
• Images published to dockerhub under odpi ie https://hub.docker.com/repository/docker/odpi/apache-ranger-admin and https://hub.docker.com/repository/docker/odpi/apache-ranger

There are some changes I hope to make

• Ensure the charts/notebooks used 'released' versions of the docker builds like 1.2 not latest
• Consider making the docker builds more fine grained (with most focus on our egeria specific ones) rather than build in a single process (except -build images)
• move out non-egeria code to a different repository - we've already done this with the connectors. For the core ranger build ideally this is independent of egeria

Now specifically the issue you found.... our build pipelines do use our own artifactory. Can you confirm where exactly you saw the above error - which image was it building? And how was the build invoked?

I've kicked off a ranger build now to check what I see locally

[RyanHolstien]

I invoked the build with the example command from the docker directory:

~ cd ./egeria/open-metadata-resources/open-metadata-deployment/docker
~ mvn -Ddocker -Ddocker.repo=odpi -Ddocker.registry=localhost:5000 clean install

Per the README.md instructions in that directory. The image it was building was the Hive plugin in the Ranger build. Here's the error from the build log, sorry about the formatting that's how it was output into my terminal.

[INFO] [INFO] -----
[INFO] -------------------------------------------------------------------
[INFO] [INFO] ------------------------------------------------
[INFO] ------------------------
[INFO] [INFO] Reacto
[INFO] r Summary for ranger 1.2.0:
[INFO] [
[INFO] INFO] 
[INFO] [INFO] ranger ............................................. SUCCESS [ 35.
[INFO] 409 s]
[INFO] [INFO] Jdbc SQL Connector ..............................
[INFO] ... SUCCESS [  5.496 s]
[INFO] [INFO] Crede
[INFO] ntial Support ................................. SUCCESS [ 26.443 s]
[INFO] [INFO] Audit Com
[INFO] ponent .................................... SUCCESS [ 17.868 s]
[INFO] [INFO] Common library for Plug
[INFO] ins ......................... SUCCESS [ 12.697 s]
[INFO] [INFO] Installer Support Component ..................
[INFO] ...... SUCCESS [  1.503 s]
[INFO] [INFO] Credential Builder ................................. SUCCESS [
[INFO]   3.985 s]
[INFO] [INFO] Embedded Web Server Invoker ........................ SUCCESS [  5.168 s]
[INFO] [INFO] Key Management Serv
[INFO] ice ............................. SUCCESS [ 20.582 s]
[INFO] [INFO] ranger-plugin-classloader ......
[INFO] .................... SUCCESS [  2.511 s]
[INFO] [INFO] HBase Security Plugin Shim ......................... SUCCESS [ 23.462 s]
[INFO] [INFO] HBase Security Plugin ....................
[INFO] .......... SUCCESS [ 11.268 s]
[INFO] [INFO] Hdfs Securit
[INFO] y Plugin ............................... SUCCESS [ 12.376 s]
[INFO] [INFO] Hive Security Plugin .............................
[INFO] .. FAILURE [03:08 min]
[INFO] [INFO] Knox Security Plugin Shi
[INFO] m .......................... SKIPPED
[INFO] [INFO] Knox S
[INFO] ecurity Plugin ............................... SKIPPED
[INFO] [INFO] Storm Security Plugin ............
[INFO] .................. SKIPPED
[INFO] [INFO] YARN
[INFO]  Security Plugin ........................
[INFO] ....... SKIPPED
[INFO] [INFO] Ranger Util ................
[INFO] ........................ SKIPPED
[INFO] [INFO] Unix Authentication Client .......
[INFO] .................. SKIPPED
[INFO] [INFO] Security Admin Web Application ..................... SKIPPED
[INFO] [IN
[INFO] FO] KAFKA Security Plugin ...............
[INFO] ............... SKIPPED
[INFO] [INFO] SOLR Security Plugin ...................
[INFO] ............ SKIPPED
[INFO] [INFO] NiFi Security Plugin ............................... SKIPPED
[INFO] [INFO] NiFi Registry Security Plugin ...
[INFO] ................... SKIPPED
[INFO] [INFO] Unix User Group Synchronizer ....................... SKIPPED
[INFO] [INFO] Ldap Config Check Tool .....
[INFO] ........................ SKIPPED
[INFO] [INFO] Unix Auth
[INFO] entication Service ........................ SKIPPED
[INFO] [INFO] KMS Security Plugin 
[INFO] ................................ SKIPPED
[INFO] [INFO] Tag Synchronizer ....................
[INFO] ............... SKIPPED
[INFO] [INFO] Hdfs Security Plugin Shim ..........................
[INFO]  SKIPPED
[INFO] [INFO] Hive Security Plugin Shim .......................... SKIPPED
[INFO] [INFO] YARN Security Plugin Shim .....................
[INFO] ..... SKIPPED
[INFO] [INFO] Storm Security Plugin shim 
[INFO] ......................... SKIPPED
[INFO] [INFO] KAFKA Security Plugin Shim .....
[INFO] .................... SKIPPED
[INFO] [INFO] SOLR Security Plugin Shim ....
[INFO] ...................... SKIPPED
[INFO] [
[INFO] INFO] Atlas Security Plugin Shim ......................... SK
[INFO] IPPED
[INFO] [INFO] KMS Security Plugin
[INFO]  Shim ........................... SKI
[INFO] PPED
[INFO] [INFO] ranger-examples .................................... SKIPPED
[INFO] [INFO] Ranger Examples 
[INFO] - Conditions and ContextEnrichers .. SKIPPED
[INFO] [INFO] Ran
[INFO] ger Examples - SampleApp .....................
[INFO] ... SKIPPED
[INFO] [INFO] Ranger Examples - Ranger Plugin for SampleApp
[INFO]  ...... SKIPPED
[INFO] [IN
[INFO] FO] Ranger Tools ....................................... SKIPPED
[INFO] [
[INFO] INFO] Atlas Security Plugin .................
[INFO] ............. SKIPPED
[INFO] [INFO] Sqoop Security Plugin .............................. SKIPPED
[INFO] [INFO] Sqoop Security Plugin Shim ......
[INFO] ................... SKIPPED
[INFO] [INFO] Kylin Security Plugin ......................
[INFO] ........ SKIPPED
[INFO] [INFO] Kylin Security Plugin Shim .....................
[INFO] .... SKIPPED
[INFO] [INFO] Unix Native Authenticato
[INFO] r .......................... SKIPPED
[INFO] [INFO] -------------------------
[INFO] -----------------------------------------------
[INFO] [INFO] BUILD FAILURE
[INFO] [INFO] ----------------------------
[INFO] --------------------------------------------
[INFO] [INFO] Total tim
[INFO] e:  06:15 min
[INFO] [INFO] Finished at: 2019-12-10T19:39:39Z
[INFO] [INFO] ----------------------------------------
[INFO] --------------------------------
[INFO] [WARNING] The requested profile "all" could not be activated because it d
[INFO] oes not exist.
[INFO] [ERROR] Failed to execute goal on project ranger-hive-plugin: Could not resolve dependencie
[INFO] s for project org.apache.ranger:ranger-hive-plugin:jar:1.2.0: Could not find artifact org.pentaho:pen
[INFO] taho-aggdesigner-algorithm:jar:5.
[INFO] 1.5-jhyde in odpi-snapshots (https
[INFO] ://odpi.jfrog.i
[INFO] o/odpi/egeria-snapshot/) -> [He
[INFO] lp 1]
[INFO] [ERROR] 
[INFO] [ERROR] To see the full stack trace
[INFO]  of the errors, re-run Maven with the 
[INFO] -e switch.
[INFO] [ERROR] Re-run Maven using the -X switch to enabl
[INFO] e full debug logging.
[INFO] [
[INFO] ERROR] 
[INFO] [ERROR
[INFO] ] For more information about 
[INFO] the errors and possible so
[INFO] lutions, please read the 
[INFO] following articles:
[INFO] [ERROR] 
[INFO] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[INFO] [ERROR] 
[INFO] [ERROR] After correcting the problems, you can resume the build with the command
[INFO] [ERROR]   mvn <goals> -rf :ranger-hive-plugin
[ERROR] The command '/bin/sh -c mvn -Pall -DskipTests=true clean compile package install assembly:assembly' returned a non-zero code: 1
[WARNING] An attempt failed, will retry 1 more times
org.apache.maven.plugin.MojoExecutionException: Could not build image
    at com.spotify.plugin.dockerfile.BuildMojo.buildImage (BuildMojo.java:247)
    at com.spotify.plugin.dockerfile.BuildMojo.execute (BuildMojo.java:135)
    at com.spotify.plugin.dockerfile.AbstractDockerMojo.tryExecute (AbstractDockerMojo.java:265)
    at com.spotify.plugin.dockerfile.AbstractDockerMojo.execute (AbstractDockerMojo.java:254)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: com.spotify.docker.client.exceptions.DockerException: The command '/bin/sh -c mvn -Pall -DskipTests=true clean compile package install assembly:assembly' returned a non-zero code: 1
    at com.spotify.plugin.dockerfile.LoggingProgressHandler.handleError (LoggingProgressHandler.java:105)
    at com.spotify.plugin.dockerfile.LoggingProgressHandler.progress (LoggingProgressHandler.java:63)
    at com.spotify.docker.client.DefaultDockerClient$BuildProgressHandler.progress (DefaultDockerClient.java:312)
    at com.spotify.docker.client.ProgressStream.tail (ProgressStream.java:77)
    at com.spotify.docker.client.DefaultDockerClient$ResponseTailReader.call (DefaultDockerClient.java:2754)
    at com.spotify.docker.client.DefaultDockerClient$ResponseTailReader.call (DefaultDockerClient.java:2738)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
    at java.lang.Thread.run (Thread.java:834)

And the reactor summary for maven:

[INFO] Reactor Summary for Open Metadata Docker Images 1.2:
[INFO] 
[INFO] Open Metadata Docker Images ........................ SUCCESS [  2.664 s]
[INFO] Docker build-time images ........................... SUCCESS [  0.053 s]
[INFO] Docker build-time images for Apache ................ SUCCESS [  0.054 s]
[INFO] Docker build-time image - Apache Atlas ............. SUCCESS [  4.249 s]
[INFO] Docker build-time image - Apache Ranger ............ FAILURE [10:54 min]
[INFO] Docker runtime image - Apache Atlas ................ SKIPPED
[INFO] Docker runtime image - Apache Ranger (admin) ....... SKIPPED
[INFO] Docker image - Configuration ....................... SKIPPED
[INFO] Docker image - Egeria .............................. SKIPPED
[INFO] Docker image - Gaian ............................... SKIPPED
[INFO] Docker image - OMRS Monitor ........................ SKIPPED

Let me know if there's anything else I can provide :)

[planetf1]

Thanks. I did reproduce the build failure. will investigate.

I should point out that the build image will rarely change (it's just ranger 1.2), also we only use in the vdc helm chart & it's not actually configured. So whilst there's no question it should work, you could skip it for now.

[planetf1]

See #958

[planetf1]

This is fixed now