syndesis-keycloak-1-prxjs pod fails to start

[grgrzybek]

When running https://syndesis.io/quickstart/ on Openshift 3.6.0 cluster created using oc cluster up, I had problems with keycloak pod. In the logs I see:

Added 'admin' to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
=========================================================================
  JBoss Bootstrap Environment
  JBOSS_HOME: /opt/jboss/keycloak
  JAVA: /usr/lib/jvm/java/bin/java
  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
06:57:43,001 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.1.Final
06:57:43,146 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
06:57:43,204 INFO  [org.jboss.as] (MSC service thread 1-4) WFLYSRV0049: Keycloak 2.5.4.Final (WildFly Core 2.0.10.Final) starting
06:57:43,477 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0055: Caught exception during boot: org.jboss.as.controller.persistence.ConfigurationPersistenceException: WFLYCTL0085: Failed to parse configuration
    at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:131) [wildfly-controller-2.0.10.Final.jar:2.0.10.Final]
    at org.jboss.as.server.ServerService.boot(ServerService.java:356) [wildfly-server-2.0.10.Final.jar:2.0.10.Final]
    at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299) [wildfly-controller-2.0.10.Final.jar:2.0.10.Final]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
Caused by: java.io.FileNotFoundException: /opt/jboss/keycloak/standalone/configuration/standalone.xml (Permission denied)
    at java.io.FileInputStream.open0(Native Method) [rt.jar:1.8.0_111]
    at java.io.FileInputStream.open(FileInputStream.java:195) [rt.jar:1.8.0_111]
    at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.8.0_111]
    at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:119) [wildfly-controller-2.0.10.Final.jar:2.0.10.Final]
    ... 3 more
06:57:43,478 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
06:57:43,481 INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
06:57:43,495 INFO  [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: Keycloak 2.5.4.Final (WildFly Core 2.0.10.Final) stopped in 12ms

digging into container I see:

sh-4.2$ pwd                                                                     
/opt/jboss/keycloak/standalone/configuration                                    
sh-4.2$ ls -la                                                                  
ls: cannot access standalone.xml: Permission denied                             
ls: cannot access syndesis-realm.json: Permission denied                        
total 48                                                                        
drwxrwx---. 3 jboss root         248 Sep 21 06:30 .                             
drwxrwx---. 8 jboss root          91 Mar  2  2017 ..                            
-rw-rw----. 1 jboss root         711 Feb 22  2017 application-roles.properties  
-rw-rw----. 1 jboss root         935 Feb 22  2017 application-users.properties  
-rw-rw----. 1 jboss root        2692 Mar  2  2017 logging.properties            
-rw-rw----. 1 jboss root         669 Feb 22  2017 mgmt-groups.properties        
-rw-rw----. 1 jboss root        1112 Feb 22  2017 mgmt-users.properties         
-rw-rw----. 1 jboss root       27040 Feb 22  2017 standalone-ha.xml             
-?????????? ? ?     ?              ?            ? standalone.xml                
-?????????? ? ?     ?              ?            ? syndesis-realm.json           
drwxrwsrwx. 2 root  1000050000    38 Sep 21 06:30 tls  

That's my first time with syndesis/opensfhit, so forgive my lack of knowledge about what should I check first...

[grgrzybek]

docker inspect showed me these mounts:

{
    "Source": "/var/lib/origin/openshift.local.volumes/pods/57753e8f-9e9b-11e7-a0c9-525400a13465/volumes/kubernetes.io~configmap/syndesis-keycloak-config/config/standalone.xml",
    "Destination": "/opt/jboss/keycloak/standalone/configuration/standalone.xml",
    "Mode": "Z",
    "RW": true,
    "Propagation": "rprivate"
},
{
    "Source": "/var/lib/origin/openshift.local.volumes/pods/57753e8f-9e9b-11e7-a0c9-525400a13465/volumes/kubernetes.io~configmap/syndesis-keycloak-config/config/syndesis-realm.json",
    "Destination": "/opt/jboss/keycloak/standalone/configuration/syndesis-realm.json",
    "Mode": "Z",
    "RW": true,
    "Propagation": "rprivate"
},

and I could see proper host versions of the mounted files.

[rhuss]

Its an issue with Docker and symbolic links into volumes when using subpath mounts. Forgot the link to the issue, but @jimmidyson knows.

As a workaround we are going to copy the mounted files into place with an init container soon.

[grgrzybek]

Thanks for info. The strange thing is that it worked on minishift machine - probably different docker version. On RHEL 7.4 I have docker 1.12.6.

[rhuss]

It works for me on minishift, too.

Ah, found the bug again --> https://bugzilla.redhat.com/show_bug.cgi?id=1481617