rabbitmq fails in OpenShift on permission issues

[surajssd]

this container image registry.centos.org/centos/rabbitmq is not built to be suitably run on OpenShift. It fails as shown below:

$ oc logs broker-1-okxyd
/run-rabbitmq-server.sh: line 6: /etc/rabbitmq/rabbitmq.config: Permission denied
set default user = guest and default password = guest
touch: cannot touch '/.run-rabbitmq-server-firstrun': Permission denied
/usr/sbin/rabbitmq-server: line 28: cd: /var/lib/rabbitmq: Permission denied
mkdir: cannot create directory '/var/lib/rabbitmq': Permission denied
Only root or rabbitmq should run rabbitmq-server

The configs used to run this on openshift are as follows:

$ cat broker-deploymentconfig.yaml 
apiVersion: v1
kind: DeploymentConfig
metadata:
  creationTimestamp: null
  labels:
    service: broker
  name: broker
spec:
  replicas: 1
  selector:
    service: broker
  strategy:
    resources: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        service: broker
    spec:
      containers:
      - env:
        - name: RABBITMQ_PASS
          value: guest
        - name: RABBITMQ_USER
          value: guest
        image: ' '
        name: coreapi-broker
        ports:
        - containerPort: 5672
          protocol: TCP
        resources: {}
      restartPolicy: Always
  test: false
  triggers:
  - type: ConfigChange
  - imageChangeParams:
      automatic: true
      containerNames:
      - coreapi-broker
      from:
        kind: ImageStreamTag
        name: broker:latest
    type: ImageChange
status: {}

and

$ cat broker-imagestream.yaml 
apiVersion: v1
kind: ImageStream
metadata:
  creationTimestamp: null
  name: broker
spec:
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: registry.centos.org/centos/rabbitmq
    generation: null
    importPolicy: {}
    name: latest
status:
  dockerImageRepository: ""

and the openshift version:

$ oc version
oc v1.4.0-rc1+b4e0954
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.17.133.24:8443
openshift v1.4.0-rc1+b4e0954
kubernetes v1.4.0+776c994

[scara]

Mostly because of OpenShift runs the container with a random user (uid) with fixed gid (0) and HOME=/.

[mohammedzee1000]

Maybe doing a chgrp -R 0 and chmod -R ug+rw for /var/lib/rabbitmq and /etc/rabbitmq during build may get it to work @scara

[mohammedzee1000]

I believe the sclorg guys have gotten a number of containers to be able to run on openshift such as https://github.com/sclorg/postgresql-container ping @hhorak

[scara]

@mohammedzee1000, yes it should work but, depending on the original package, nss_wrapper could be required too. Theory in https://docs.openshift.com/enterprise/3.1/creating_images/guidelines.html#use-uid. Example: https://github.com/sclorg/postgresql-container/commit/1916d06fab2f4e04c7511beaa17c7d6480a0d4d3.

[mohammedzee1000]

All right did some circus with the nss_wrapper but now i am facing this error Here is where i am working on it https://github.com/mohammedzee1000/CentOS-Dockerfiles/tree/2017-01-18_12-14-12-rabbitmq_openshift/rabbitmq/centos7 Log from an oc new-app run

set default user = rabbitmq and default password = guest
{error_logger,{{2017,1,18},{13,42,18}},"Failed to create cookie file '/.erlang.cookie': eacces",[]}
{error_logger,{{2017,1,18},{13,42,18}},crash_report,[[{initial_call,{auth,init,['Argument__1']}},{pid,<0.20.0>},{registered_name,[]},{error_info,{exit,{"Failed to create cookie file '/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,285}]},{auth,init,1,[{file,"auth.erl"},{line,139}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[<0.18.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,941}],[]]}
{error_logger,{{2017,1,18},{13,42,18}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{"Failed to create cookie file '/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,285}]},{auth,init,1,[{file,"auth.erl"},{line,139}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{offender,[{pid,undefined},{name,auth},{mfargs,{auth,start_link,[]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2017,1,18},{13,42,18}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,285}]},{auth,init,1,[{file,"auth.erl"},{line,139}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2017,1,18},{13,42,18}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,285}]},{auth,init,1,[{file,"auth.erl"},{line,139}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,150}],[]]}
{error_logger,{{2017,1,18},{13,42,18}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,285}]},{auth,init,1,[{file,"auth.erl"},{line,139}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{\"Failed to create cookie file '/.erlang.cookie': eacces\",[{auth,init_cookie,0,[{file,\"auth.erl\"},{line,285}]},{auth,init,1,[{file,\"auth.erl\"},{line,139}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}}}}},{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/.er

[mohammedzee1000]

Ok found the issue. Inspite of me forcing the users home directory in the passwd.template, it still shows $HOME as /

@scara thoughts?

[scara]

Try to define it by exporting the new value e.g. https://github.com/sclorg/postgresql-container/commit/1916d06fab2f4e04c7511beaa17c7d6480a0d4d3#diff-9ec6b0d09f3886cdcc98deededdfced8R4.

[mohammedzee1000]

@scara thanks that and a little wisdom i picked up from a another container got me

$ oc new-app -e RABBITMQ_USER=rabbitmq -e RABBITMQ_PASS=guest mohammedzee1000/testc

$ oc logs testc-1-1wjkb
set default user = rabbitmq and default password = guest

              RabbitMQ 3.3.5. Copyright (C) 2007-2014 GoPivotal, Inc.
  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
  ##  ##
  ##########  Logs: -
  ######  ##        -
  ##########
              Starting broker... completed with 6 plugins.

[mohammedzee1000]

@scara you are welcome to validate my first work with nss_wrapper :)

[scara]

Yeah 😃! Great work 👍.

Edit: given that you've covered what theory and example were suggesting, you could just squash the two commits into a single one, editing the commit message too. Unfortunately I'm not in the position of being a better peer reviewer since I'm new to OpenShift. Let's wait for someone more qualified than me 😉.

HTH, Matteo

[concaf]

Okay, so #118 does not fix this for me.

After some hours of confusion, I found out that the newer image resulting from #118 worked with oc new-app because in the newer Dockerfile VOLUME /var/lib/rabbitmq is specified, and oc new-app detects that there is a volume declared by the image, and hence creates a host volume on that pod.

<snip>
        volumeMounts:                                                                              
        - mountPath: /var/lib/rabbitmq                                                               
          name: rabbitmq-volume-1
<snip>
      volumes:
      - emptyDir: {}
        name: rabbitmq-volume-1                                                            

And due to this volume mount, the permissions are all fixed. However, if we are not using volume mounts, the same errors are outputted on the screen. There is no effect of the nss_wrapper and changing the permissions. If we added volumes in the older image, it would still work.

Maybe try this with - oc run NAME --image=image and see if it works instead of oc new-app

[mohammedzee1000]

Thanks, i will see what i can do. The issue seems to be with /var/lib/rabbitmq, and not the other directories. Also, the only directory that should ideally be shared is /var/lib/rabbitmq as that is where the data is dumped by rabbitmq afaik

[mohammedzee1000]

Container updated. It appears the volume mount was playing the havoc, removed in https://github.com/CentOS/CentOS-Dockerfiles/pull/120

Tested as working

$ oc run test1 --image=mohammedzee1000/rabbitmq_test1 --env="RABBITMQ_USER=rabbitmq" --env="RABBITMQ_PASS=guest"

$ oc get pods
NAME            READY     STATUS    RESTARTS   AGE
test1-1-aliqd   1/1       Running   0          36s

$ oc logs test1-1-aliqd
set default user = rabbitmq and default password = guest

              RabbitMQ 3.3.5. Copyright (C) 2007-2014 GoPivotal, Inc.
  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
  ##  ##
  ##########  Logs: -
  ######  ##        -
  ##########
              Starting broker... completed with 6 plugins.