built-in support for glusterfs client on worker nodes

[m-usmanayub]

We are implementing glusterfs cluster to be used as storage for persistent volumes using Heketi API and StorageClasses.

The scenario works only when the glusterfs client package is installed on the worker nodes manually when it comes to pod mounting. Since the manual package installation can be overwritten anytime the reconciler is run, it is requested to make it part of worker node image so that it does not have to be installed manually and also works in autoscale scenario.

[timoreimann]

Hey @m-usmanayub 👋

for cases like this, we usually recommend running a Daemonset that installs the binary onto the host system. One key advantage of this approach is that you do not depend on the cloud provider and can update the binary per your own preferences.

How do you feel about this approach?

[timoreimann]

(Sorry for the accidental close and half-finished comment, I fat-fingered that one.)

[m-usmanayub]

Hey @m-usmanayub 👋

for cases like this, we usually recommend running a Daemonset that installs the binary onto the host system. One key advantage of this approach is that you do not depend on the cloud provider and can update the binary per your own preferences.

How do you feel about this approach?

Hi @timoreimann

The approach seems realistic. Do you have any example of this?

Also in such a case we need to know the OS flavor (Ubuntu, CentOS etc) running on the worker nodes and install the relevant OS Package (deb/rpm) by creating a customized docker image. am I right?

[timoreimann]

@m-usmanayub I'd try to create a Daemonset that mounts the root host-filesystem, chroots into it, checks if the package is already installed, and if not updates the repository and installs the package. We currently run Debian, so the logic could look something like this (untested):

#!/bin/bash

chroot /host /bin/bash -u <<'EOF'
export GLUSTERFS_PACKAGE=glusterfs-client

dpkg -l ${GLUSTERFS_PACKAGE} > /dev/null
case $? in
  0)
    echo "nothing to do: glusterfs package is already installed"
    ;;
  1)
    echo "installing glusterfs package"
    apt-get update && apt-get install -y ${GLUSTERFS_PACKAGE}
    ;;
  *)
    echo "failed to check if glusterfs is installed" >&2
    exit 1
    ;;
esac

sleep infinity
EOF

You might not even need a custom container image if you choose to inline the script as arguments to a container that comes with bash (e.g., a Debian-based image).

For mounting the host file-system, you'll need your Daemonset to specify volume and volume mount fields accordingly. A small blog post on the topic is available here, and our own doks-debug Daemonset does it as well. (Adjust the manifest to your needs, of course.)

Hope this gets you started. Note that while we do not have plans to do so, we could change the distro and/or distro version in the future. Such a change would likely be bundled with a Kubernetes minor release, however, so if you keep an eye on the release notes you should be able to respond to any updates we might do that affect how your Daemonset works.

[m-usmanayub]

@timoreimann

Thanks for the explanation, I was able to run a DaemonSet using the logic mentioned however with some changes. As you said we would have to keep a check on the distro and its version of the worker nodes. Please feel free to close the issue if you believe that the package cannot be/will never be made part of the worker nodes. Thanks.

I am posting the yaml definition for anyone who would like to use.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gfs-client
  namespace: kube-system
  labels:
    app: gfs-client
spec:
  selector:
    matchLabels:
      name: gfs-client
  template:
    metadata:
      labels:
        name: gfs-client
    spec:
      hostNetwork: true
      containers:
      - name: ubuntu
        securityContext:
          privileged: true
        image: ubuntu:bionic
        args:
        - /bin/bash
        - "-c"
        - |
          chroot /host /bin/bash -u <<'EOF'
          export GLUSTERFS_PACKAGE=glusterfs-client
          apt list --installed | grep ${GLUSTERFS_PACKAGE} > /dev/null
          case $? in
            0)
              echo "nothing to do: glusterfs package is already installed"
            ;;
            1)
              echo "installing glusterfs package"
              apt update && apt install -y ${GLUSTERFS_PACKAGE}
            ;;
            *)
              echo "failed to check if glusterfs is installed" >&2
              exit 1
             ;;
          esac
          sleep infinity
          EOF
        volumeMounts:
          - name: host
            mountPath: /host
          - name: docker
            mountPath: /var/run/docker.sock
      priorityClassName: system-cluster-critical
      volumes:
        - name: host
          hostPath:
            path: /
        - name: docker
          hostPath:
            path: /var/run/docker.sock
            type: Socket

[filidorwiese]

A very nice solution! In the spirit of sharing, I've updated the installation script above :point_up: to install a more recent version of glusterfs-client. At the time of writing, the master nodes are based on Debian Buster which are shipped with version 5.5-3. Due to some strange and persistent issues with missing files / non-empty directories showing as empty on the gluster volumes, I had to update both the server and the client to 8.3-1 from upstream. Here's my code:

image: ubuntu:groovy
args:
  - /bin/bash
  - "-c"
  - |
    chroot /host /bin/bash -u <<'EOF'
    export GLUSTERFS_PACKAGE=glusterfs-client
    apt show ${GLUSTERFS_PACKAGE}
    apt list --installed | grep ${GLUSTERFS_PACKAGE} > /dev/null
    case $? in
      0)
        echo "nothing to do: glusterfs package is already installed"
      ;;
      1)
        echo "installing glusterfs package"
        wget -O - https://download.gluster.org/pub/gluster/glusterfs/7/rsa.pub | apt-key add -
        DEBID=$(grep 'VERSION_ID=' /etc/os-release | cut -d '=' -f 2 | tr -d '"')
        DEBVER=$(grep 'VERSION=' /etc/os-release | grep -Eo '[a-z]+')
        DEBARCH=$(dpkg --print-architecture)
        echo deb https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/${DEBID}/${DEBARCH}/apt ${DEBVER} main > /etc/apt/sources.list.d/gluster.list
        apt update && apt install -y ${GLUSTERFS_PACKAGE}
      ;;
      *)
        echo "failed to check if glusterfs is installed" >&2
        exit 1
       ;;
    esac
    sleep infinity
    EOF

[timoreimann]

I'll be closing out this ticket given a workaround is possible. I could imagine a future feature where customers can choose from a limited list of packages / node properties to enable on bootstrap. This needs broader discussion, however, and consideration for how DOKS worker node images may possibly evolve. We'll continue to track this possibility internally.