Runing jenkins with /var/jenkins_home on longhorn volume without root failed

[amioranza]

Hi guys!

I'm trying to deploy an scalable jenkins on kubernetes using a persistent volume from longhorn to keep /var/jenkins_home.

This is my deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: jenkins
  name: jenkins-master
spec:
  selector:
    matchLabels:
      app: jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
      - env:
        - name: JAVA_OPTS
          value: -Djenkins.install.runSetupWizard=false
        image: jenkins:2.138-1
        imagePullPolicy: Always
        name: jenkins
        ports:
        - containerPort: 8080
          name: http-port
          protocol: TCP
        - containerPort: 50000
          name: jnlp-port
          protocol: TCP

        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvc

This is my pvc file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

This is the Dockerfile to generate the custom image:

FROM jenkins/jenkins:2.138
# Plugins
RUN /usr/local/bin/install-plugins.sh active-directory \
ssh-slaves \
ssh \
ssh-agent \
pipeline-utility-steps \
kubernetes-pipeline-aggregator \
gitlab-plugin \
docker-plugin \
email-ext \
mailer \
htmlpublisher \
cucumber-reports \
blueocean \
greenballs \
simple-theme-plugin \
config-file-provider \
icon-shim \
jdk-tool \
matrix-auth \
thinBackup \
timestamper \
pipeline-maven \
kubernetes 
USER root
RUN apt-get update && apt-get install -y sudo maven
RUN usermod -G sudo jenkins
RUN chown -R jenkins:jenkins /var/jenkins_home
USER jenkins

The pvc uses the longhorn StorageClass created by the oficial helm deployment. When I submit the pvc it works as expected and create a new PV with 10Gi.

The problem happens with the deployment, when I submit the the deployment the jenkins container failed to start with this error message:

touch: cannot touch '/var/jenkins_home/copy_reference_file.log': Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?

As a good practice the user running jenkins app is a user jenkins. This user don't have permission to write to /var/jenkins_home, but the Dockerfile execute a chown on the /var/jenkins_home to the user and group jenkins, but the longhorn doesn't respect it. I cannot find where I can specify other user than root to access longhorn volumes.

I've changed the Dockerfile to make a sleep and allow me to access the container to check the permissions, this is the final permissions:

jenkins@jenkins-master-7db565fdf8-5c4m2:/$ ls -la /var
total 48
drwxr-xr-x 32 root root  4096 Sep 18 12:23 .
drwxr-xr-x 79 root root  4096 Sep 18 16:26 ..
drwxr-xr-x  2 root root  4096 Jun 26 12:03 backups
drwxr-xr-x 21 root root  4096 Sep 18 12:23 cache
drwxr-xr-x  3 root root  4096 Sep 18 13:13 jenkins_home
drwxr-xr-x 22 root root  4096 Sep 18 12:25 lib
drwxrwsr-x  2 root staff 4096 Jun 26 12:03 local
lrwxrwxrwx  1 root root     9 Jul 16 00:00 lock -> /run/lock
drwxr-xr-x  9 root root  4096 Sep 18 12:26 log
drwxrwsr-x  2 root mail  4096 Jul 16 00:00 mail
drwxr-xr-x  2 root root  4096 Jul 16 00:00 opt
lrwxrwxrwx  1 root root     4 Jul 16 00:00 run -> /run
drwxr-xr-x  2 root root  4096 Jul 16 00:00 spool
drwxrwxrwt  2 root root  4096 Jun 26 12:03 tmp
jenkins@jenkins-master-7db565fdf8-5c4m2:/$ ls -la /var/jenkins_home
total 24
drwxr-xr-x  3 root root  4096 Sep 18 13:13 .
drwxr-xr-x 32 root root  4096 Sep 18 12:23 ..
drwx------  2 root root 16384 Sep 18 13:13 lost+found

There is a way to apply permissions for other users than root to access and make changes on longhorn mounts?

Thanks.

[amioranza]

Hi,

This is not a longhorn problem, it is a security context mssing on the deployment, we fixed it with the options below:

...
    spec:
      serviceAccountName: jenkins
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - env:
...

Thanks.