Clarify Kubernetes plugin volume mounting process

Hello! I'm trying to mount a persistent volume that will eventually contain plugin jar files. Graylog really hates the lost+found directory created in Kubernetes volume mounts, but I worked around that for the Journal by telling Graylog to use a subdirectory of the volume.

To no surprise, mounting the volume to /usr/share/graylog/plugin has the container in CrashLoopBackoff, saying:

find: '/usr/share/graylog/plugin/lost+found': Permission denied

The issue is, in my configuration, I have an environment variable telling Graylog to search a subdirectory of the volume for plugins, much like the Journal:

# excerpt from https://gitlab.com/door-to-darkness/kustomize-logging-stack/-/blob/18791ebc8f9a2b771822a90e442195b954580170/components/graylog/deployment.yaml
          env:
            - name: GRAYLOG_MESSAGE_JOURNAL_DIR
              value: data/journal/graylog
            - name: GRAYLOG_PLUGIN_DIR
              value: plugin/graylog
          envFrom:
            - configMapRef:
                name: graylog-config-env
# ... abbreviated ...
            - name: journal
              mountPath: /usr/share/graylog/data/journal
            - name: plugins
              mountPath: /usr/share/graylog/plugin

https://github.com/Graylog2/graylog-docker/issues/133 originally raised the issue of supporting multiple plugin paths and PR https://github.com/Graylog2/graylog-docker/pull/148 added behavior to the container's entrypoint script to merge a user provided plugins directory with Graylog's default plugins:

# Merge plugin dirs to allow mounting of /plugin as a volume
export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
rm -f /usr/share/graylog/plugins-merged/*
find /usr/share/graylog/plugins-default/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;
find /usr/share/graylog/plugin/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;

Based on that configuration, there's no way to tell Graylog where to look for the user's plugins before merging, as the paths are hardcoded and GRAYLOG_PLUGIN_DIR is (rightfully) overridden for the Graylog process. It seems to me that the entrypoint should do something more like:

# Merge plugin dirs to allow mounting of /plugin as a volume
rm -f ${GRAYLOG_HOME}/plugins-merged/*
find ${GRAYLOG_HOME}/plugins-default/ -type f -exec cp {} ${GRAYLOG_HOME}/plugins-merged/ \;
find ${GRAYLOG_HOME}/${GRAYLOG_PLUGIN_DIR}/ -type f -exec cp {} ${GRAYLOG_HOME}/plugins-merged/ \;
export GRAYLOG_PLUGIN_DIR=${GRAYLOG_HOME}/plugins-merged

The above isn't a full solution though because it doesn't handle the case of GRAYLOG_PLUGIN_DIR being an absolute path, as is stated as an option in https://docs.graylog.org/docs/server-conf

I'm opening this issue because I think the entrypoint script could be improved to better handle Kubernetes based deployments, but I could be fundamentally misunderstanding how Graylog wants me to load plugins in Kubernetes. In that case, I'm hoping to get some insight into the right way to get this done and hopefully update the documentation to reflect that.

Graylog2 / graylog-docker

Clarify Kubernetes plugin volume mounting process #206