search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.96k stars 9.19k forks source link

[bitnami/fluentd] Unreadable log mounts #1905

Closed austince closed 4 years ago

austince commented 4 years ago

Which chart:

Fluentd 0.4.10

Description

Mounted logs are unreadable. It seems similar to this issue: https://stackoverflow.com/questions/51671212/fluentd-log-unreadable-it-is-excluded-and-would-be-examined-next-time

I have already set the FLUENT_UID to '0'. What other information do you need to find out what else I could be doing wrong?

Steps to reproduce the issue:

  1. Deploy the chart on EKS
  2. Tail the logs of the daemonset
  3. See error

Describe the results you received:

2020-02-11 00:28:26 +0000 [warn]: #0 /var/log/containers/prometheus-prometheus-operator-prometheus-1_monitoring_prometheus-config-reloader-5de238447ab71c5f5e77c3890822c9fe5e0caefb212003488747ba73d4a0e603.log unreadable. It is excluded and would be examined next time.

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Version of Helm and Kubernetes:

Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-21T22:17:28Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
austince commented 4 years ago

I have another Fluentd daemonset running as well - could they be interfering with one another?

javsalgar commented 4 years ago

Hi,

Did you deploy it with the security group as root? 

      {{- if .Values.forwarder.securityContext.enabled }}
      securityContext:
        runAsUser: {{ .Values.forwarder.securityContext.runAsUser }}
        fsGroup: {{ .Values.forwarder.securityContext.fsGroup }}
austince commented 4 years ago

Hey @javsalgar, yeah here's my current config:

forwarder:
    ## Enable forwarder daemonset
    ##
    enabled: true

    ## K8s Security Context for forwarder pods
    ## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
    ##
    securityContext:
      enabled: true
      runAsUser: 0
      fsGroup: 0
austince commented 4 years ago

Looks like it's being mounted as rw, if I'm reading this describe output correctly: 

    Mounts:
      /opt/bitnami/fluentd/conf from fluentd-config (rw)
      /opt/bitnami/fluentd/logs/buffers from buffer (rw)
      /var/lib/docker/containers from varlibdockercontainers (rw)
      /var/log from varlog (rw)
austince commented 4 years ago

I've also verified that whoami prints root, and that the files are readable. Could it be something to do with symlinks? Everything in the /var/log/containers directory is a symlink.

lrwxrwxrwx 1 root root  121 Jan  5 01:59 weave-scope-agent-weave-scope-6ps7d_monitoring_weave-scope-agent-1c61350b6787932925ac9f6a35855614561103382bfedd7f374d4dd68cf96841.log -> /var/log/pods/monitoring_weave-scope-agent-weave-scope-6ps7d_f53d4ca9-2f5e-11ea-ae59-0acec118fa77/weave-scope-agent/0.log
jixuju commented 4 years ago

I have the same problem.

Error log in forwarder

# kubectl logs -f fluentd-t7vpt -n logging
2020-02-15 07:46:26 +0000 [warn]: #0 /var/log/containers/es-elasticsearch-data-0_logging_elasticsearch-b788c87b88ca72f6e6b4a9853d3d7236bed636d07c939011e21ddcc63118b84e.log unreadable. It is excluded and would be examined next time.
2020-02-15 07:46:26 +0000 [warn]: #0 /var/log/containers/es-elasticsearch-metrics-6845b86646-kbs57_logging_metrics-d5aa13ceadda11388cbc7b6e4a1995bb127cafa8a829386214f91d7bc3de01c1.log unreadable. It is excluded and would be examined next time.

Forwarder container current user

root@fluentd-t7vpt:/var/lib/docker/containers# whoami
root

Part configuration of daemonset fluent

spec:
    volumes:
    - name: fluentd-config
        configMap:
        name: fluentd-forwarder-cm
        defaultMode: 420
    - name: buffer
        emptyDir: {}
    - name: varlog
        hostPath:
        path: /var/log
        type: ''
    - name: varlibdockercontainers
        hostPath:
        path: /var/lib/docker/containers
        type: ''
    containers:
    - name: fluentd
        image: 'docker.io/bitnami/fluentd:1.9.2-debian-10-r1'
        ports:
        - name: http
            containerPort: 9880
            protocol: TCP
        env:
        - name: FLUENTD_CONF
            value: fluentd.conf
        - name: FLUENTD_OPT
        resources: {}
        volumeMounts:
        - name: fluentd-config
            mountPath: /opt/bitnami/fluentd/conf
        - name: buffer
            mountPath: /opt/bitnami/fluentd/logs/buffers
        - name: varlog
            mountPath: /var/log
        - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers

Part of the configuration of forwarder in values.yaml

forwarder:
  ## Enable forwarder daemonset
  ##
  enabled: true

  ## K8s Security Context for forwarder pods
  ## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  ##
  securityContext:
    enabled: true
    runAsUser: 0
    fsGroup: 0

  ## Name of the config file that will be used by Fluentd at launch
  ## Fluentd will look for it under the /opt/bitnami/fluentd/conf directory
  ##
  configFile: fluentd.conf

Log path relationship

lrwxrwxrwx 1 root root 102 Feb 15 10:57 es-elasticsearch-data-0_logging_elasticsearch-b788c87b88ca72f6e6b4a9853d3d7236bed636d07c939011e21ddcc63118b84e.log -> /var/log/pods/logging_es-elasticsearch-data-0_1d1096fe-c372-4ce6-8232-fbae77d0d9e1/elasticsearch/0.log

# cd /var/log/pods/logging_es-elasticsearch-data-0_1d1096fe-c372-4ce6-8232-fbae77d0d9e1/elasticsearch
# ls -lrt
total 0
lrwxrwxrwx 1 root root 165 Feb 15 10:57 0.log -> /var/lib/docker/containers/b788c87b88ca72f6e6b4a9853d3d7236bed636d07c939011e21ddcc63118b84e/b788c87b88ca72f6e6b4a9853d3d7236bed636d07c939011e21ddcc63118b84e-json.log
javsalgar commented 4 years ago

/var/log/pods/monitoring_weave-scope-agent-weave-scope-6ps7d_f53d4ca9-2f5e-11ea-ae59-0acec118fa77/weave-scope-agent/0.log

The question would be, is that path also mounted in the container? If not, the symlink would not work

jixuju commented 4 years ago

By default, /var/log/pods/...is also a symlink to /var/lib/docker/containers/..., and both of these addresses have been mounted

javsalgar commented 4 years ago

Could you also show me the permissions of the symlinked file inside the container? Are you able to read it with cat for example? I'm trying to understand the reason of not being readable

jixuju commented 4 years ago

This is the log of what I did inside the container:

root@fluentd-hb7hg:/var/log/containers# cd /var/log/containers
root@fluentd-hb7hg:/var/log/containers# ls -lrt
total 0
lrwxrwxrwx 1 root root  96 Feb 14 12:29 kube-proxy-46btn_kube-system_kube-proxy-e4713ec47c3159d78620694cf0e15a18f939a13fbb58752e1df334c3c0662445.log -> /var/log/pods/kube-system_kube-proxy-46btn_60f2005c-d970-4cab-87a8-1bec6bebed1d/kube-proxy/4.log
lrwxrwxrwx 1 root root  98 Feb 18 11:37 elastic-operator-0_elastic-system_manager-95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b.log -> /var/log/pods/elastic-system_elastic-operator-0_ae948fd8-c218-4a7b-be5b-29d796a2c7e0/manager/0.log
...

root@fluentd-hb7hg:/var/log/containers# cat elastic-operator-0_elastic-system_manager-95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b.log | more
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.870Z\",\"logger\":\"manager\",\"message\":\"Setting up client for manager\",\"
ver\":\"1.0.1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871173858Z"}
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.871Z\",\"logger\":\"manager\",\"message\":\"Setting up scheme\",\"ver\":\"1.0.
1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871284141Z"}
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.872Z\",\"logger\":\"manager\",\"message\":\"Setting up manager\",\"ver\":\"1.0
.1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.873000029Z"}
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.872Z\",\"logger\":\"manager\",\"message\":\"Operator configured to manage all 
namespaces\",\"ver\":\"1.0.1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.873023016Z"}
...

root@fluentd-hb7hg:/var/log/containers# cd /var/log/pods/elastic-system_elastic-operator-0_ae948fd8-c218-4a7b-be5b-29d796a2c7e0/manager/
root@fluentd-hb7hg:/var/log/pods/elastic-system_elastic-operator-0_ae948fd8-c218-4a7b-be5b-29d796a2c7e0/manager# ls -lrt
total 0
lrwxrwxrwx 1 root root 165 Feb 18 11:37 0.log -> /var/lib/docker/containers/95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b/95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b-json.log
root@fluentd-hb7hg:/var/log/pods/elastic-system_elastic-operator-0_ae948fd8-c218-4a7b-be5b-29d796a2c7e0/manager# cat 0.log|more
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.870Z\",\"logger\":\"manager\",\"message\":\"Setting up client for manager\",\"
ver\":\"1.0.1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871173858Z"}
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.871Z\",\"logger\":\"manager\",\"message\":\"Setting up scheme\",\"ver\":\"1.0.
1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871284141Z"}
...

root@fluentd-hb7hg:/var/log/pods/elastic-system_elastic-operator-0_ae948fd8-c218-4a7b-be5b-29d796a2c7e0/manager# cat /var/lib/docker/containers/95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b/95a0782ceb162b9c3f796e4ebf93eb06f6761496ae4a809465c22d7e369ed71b-json.log|more
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.870Z\",\"logger\":\"manager\",\"message\":\"Setting up client for manager\",\"
ver\":\"1.0.1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871173858Z"}
{"log":"{\"level\":\"info\",\"@timestamp\":\"2020-02-18T11:37:23.871Z\",\"logger\":\"manager\",\"message\":\"Setting up scheme\",\"ver\":\"1.0.
1-bcb74688\"}\n","stream":"stderr","time":"2020-02-18T11:37:23.871284141Z"}
...
r3kzi commented 4 years ago

Same here. Also with AWS EKS.

$ helm ls -n monitoring                                    
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION
fluentd monitoring  1           2020-02-19 10:59:00.164007398 +0100 CET deployed    fluentd-0.4.15  1.9.2  
2020-02-19 10:09:37 +0000 [warn]: #0 /var/log/containers/kube-proxy-sj5xx_kube-system_kube-proxy-6241050f2a5bf4cf2375cdf8e08de119b8a4b7d7b2eea4c093774ead6bb40fca.log unreadable. It is excluded and would be examined next time.
2020-02-19 10:09:37 +0000 [warn]: #0 /var/log/containers/aws-node-46thb_kube-system_aws-node-0fde4bf1ece845095fe27e145713637571eacff1e3bed2a64eb5da37c5dee995.log unreadable. It is excluded and would be examined next time.
2020-02-19 10:09:37 +0000 [warn]: #0 /var/log/containers/fluentd-0_monitoring_fluentd-548d9891320e2baec0ec4f472a586dc9ab63048f938beb9d2109ff1790da4c9c.log unreadable. It is excluded and would be examined next time.
2020-02-19 10:09:37 +0000 [warn]: #0 /var/log/containers/fluentd-fzffp_monitoring_fluentd-66a2d2fc32b1a4279afe6500f319857bee686d592245d7687812db5c645abfcf.log unreadable. It is excluded and would be examined next time.
2020-02-19 10:09:37 +0000 [warn]: #0 /var/log/containers/fluentd-fzffp_monitoring_fluentd-95202089a43c2cc7781fd4aab604fa55e5e65838460d0a4fac373cfe384abffa.log unreadable. It is excluded and would be examined next time.
$ k get pods -n monitoring fluentd-2fxbb -o yaml | grep security -A 2
  securityContext:
    fsGroup: 0
    runAsUser: 0

fluentd runs with a fluentd user

$ k exec -it -n monitoring fluentd-6p6tc  -- ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
fluentd      1  1.2  0.7 568184 55820 ?        Ssl  11:33   0:01 /opt/bitnami/ru
fluentd     64  1.3  0.7 1109952 55820 ?       Sl   11:33   0:01 /opt/bitnami/ru
root       101  0.0  0.0   7640  2748 pts/0    Rs+  11:35   0:00 ps aux

After an user switch i was not able to cat the log files.

$ k exec -it -n monitoring fluentd-6p6tc  -- bash      
root@fluentd-6p6tc:/opt/bitnami/fluentd# su fluentd
$ cat /var/log/containers/aws-node-6qjjq_kube-system_aws-node-2632215354a6145a615283f0af49a172eff42f748096e813c328d768a35e3f53.log
cat: /var/log/containers/aws-node-6qjjq_kube-system_aws-node-2632215354a6145a615283f0af49a172eff42f748096e813c328d768a35e3f53.log: Permission denied
javsalgar commented 4 years ago

Hi,

You showed me the permissions of the symlink, but not the permissions of the symlinked file.

/var/log/pods/kube-system_kube-proxy-46btn_60f2005c-d970-4cab-87a8-1bec6bebed1d/kube-proxy/4.log

It definitely seems that the permissions of these logs are restrictive so the fluentd user is unable to read form them. One option that comes to mind is to change the permissions in the host system so the logs are readable. I am not sure about forcing the fluentd process to be run as pure root.

r3kzi commented 4 years ago 
root@fluentd-75tcn:/opt/bitnami/fluentd# ls -la /var/log/containers/kube-proxy-jh64q_kube-system_kube-proxy-dd7f36d3b76eb33bb976a9c7679e0df16a808792971c9b1e7b3b3379bd4c9ec5.log 
lrwxrwxrwx 1 root root 96 Feb 19 06:47 /var/log/containers/kube-proxy-jh64q_kube-system_kube-proxy-dd7f36d3b76eb33bb976a9c7679e0df16a808792971c9b1e7b3b3379bd4c9ec5.log -> /var/log/pods/kube-system_kube-proxy-jh64q_ace56ae7-52e3-11ea-99e7-0acb4d77488c/kube-proxy/0.log

root@fluentd-75tcn:/opt/bitnami/fluentd# ls -la /var/log/pods/kube-system_kube-proxy-jh64q_ace56ae7-52e3-11ea-99e7-0acb4d77488c/kube-proxy/0.log
lrwxrwxrwx 1 root root 165 Feb 19 06:47 /var/log/pods/kube-system_kube-proxy-jh64q_ace56ae7-52e3-11ea-99e7-0acb4d77488c/kube-proxy/0.log -> /var/lib/docker/containers/dd7f36d3b76eb33bb976a9c7679e0df16a808792971c9b1e7b3b3379bd4c9ec5/dd7f36d3b76eb33bb976a9c7679e0df16a808792971c9b1e7b3b3379bd4c9ec5-json.log
javsalgar commented 4 years ago

Did the PR fix the issue?

r3kzi commented 4 years ago

Unfortunately no :-(

Same behaviour with:

$ helm ls -n monitoring
NAME                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                       APP VERSION
fluentd             monitoring  1           2020-02-20 14:56:20.357222481 +0100 CET deployed    fluentd-0.4.16              1.9.2
r3kzi commented 4 years ago

Somehow Promtail works just fine. They are doing similiar things..https://github.com/grafana/loki/blob/master/production/helm/promtail/values.yaml#L91-L105

r3kzi commented 4 years ago

I don't get it....we can't be the first that are running fluentd on AWS EKS :joy:

austince commented 4 years ago

I think it's directly related to the fluentd user not having read permissions for those files but not sure how to change that. We use the official fluent/fluentd-kubernetes-daemonset image and it works well, but runs the fluentd process as root. The bitnami image runs as user 1001. Unfortunately it's not an easy drop-in replacement as this chart relies on the file structure of the bitnami image.

r3kzi commented 4 years ago

I just saw that Promtail runs as root as well.

root@loki-promtail-cfjmb:/# cat /proc/1/status
Name:   promtail
Umask:  0022
State:  S (sleeping)
Tgid:   1
Ngid:   0
Pid:    1
PPid:   0
TracerPid:  0
Uid:    0   0   0   0
Gid:    0   0   0   0

root@fluentd-l4wlv:/opt/bitnami/fluentd# cat /proc/1/status 
Name:   fluentd
Umask:  0022
State:  S (sleeping)
Tgid:   1
Ngid:   0
Pid:    1
PPid:   0
TracerPid:  0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000

I will try using fluent/fluentd-kubernetes-daemonset. Thanks!

javsalgar commented 4 years ago

In this kind of scenarios what we could do is to allow initContainers that would change the permissions of your mounted files or, instead, change the permissions in your host system. We try to avoid using root processes as much as possible.

austince commented 4 years ago

Understandably so @javsalgar -- I'd avoid changing permissions on the host system as much as possible as well, especially in a managed k8s environment like EKS, but interested to see how the initContainers idea would work. Do you have a rough sketch?

javsalgar commented 4 years ago

The thing is that the root container would also change permissions on the host system because essentially, it would just chmod the mounted folders. I'm not 100% sure what the best way would be in order to avoid a root user running and, at the same time, to not change permissions on the host system.

ghost commented 4 years ago

Setting FLUENT_UID to 0 didn't work for me. I had to add the following to forwarder.extraEnv

  - name: FLUENTD_DAEMON_USER
    value: root
  - name: FLUENTD_DAEMON_GROUP
    value: root

Since only the root user has access to /var/lib/docker/containers on the host.

javsalgar commented 4 years ago

Thanks for sharing! I believe it would be interesting adding this to the chart documentation.

stale[bot] commented 4 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

javsalgar commented 4 years ago

Not stale

irizzant commented 4 years ago

See #2253. The workaround suggested by @brettcat is correct, it should be included by default in values.yaml and values-production.yaml or Fluentd is not able to read any log in either case.

andresbono commented 4 years ago

Just to give my point of view, executing the main process of the application as root goes against our security conventions. It is true that in certain environments, if you don't do that, fluentd is totally unusable. But it is decision of the administrator how to workaround that issue.

I wouldn't add the workaround suggested by @brettcat directly in the values*.yaml; the administrators should be conscious of what they are doing by actually applying the workaround themselves.

I'd rather add it to the chart README.md.

Thoughts @javsalgar?

irizzant commented 4 years ago

It is true that in certain environments, if you don't do that, fluentd is totally unusable. But it is decision of the administrator how to workaround that issue.

I don't see the match with certain environments at all. The values.yaml and values-production.yaml both have the following defaults:

forwarder.securityContext.enabled Enable security context for forwarder pods true
forwarder.securityContext.fsGroup Group ID for forwarder's containers filesystem 0
forwarder.securityContext.runAsUser User ID for forwarder's containers 0

as reported in the Prerequisites:

Note: Please, note that the forwarder runs the container as root by default setting the forwarder.securityContext.runAsUser to 0 (root user)

Having the above defaults implies that files mounted in volumes are only readable by root regardless from the environment you choose, so either you add the

- name: FLUENTD_DAEMON_USER
    value: root
  - name: FLUENTD_DAEMON_GROUP
    value: root

variables to be compliant with chart defaults or the chart as is will be unusable.

javsalgar commented 4 years ago

Hi,

As far as I know, I think that the volumes could be readable by other user different from root. However I'm not entirely sure if fsGroup would apply correctly to hostPath volumes. Anyways, in the chart there is an extraEnv section that would allow you to set the FLUENTD_DAEMON_USER and FLUENTD_DAEMON_GROUP. If several users find this issue and require it to be fully root, we could consider adding a value in the chart that would set these two env vars.

austince commented 4 years ago

Thanks @javsalgar - I think we have the several we need to make that change:

irizzant commented 4 years ago

Exactly, this is not user related either.

As reported here hostPath volumes are only writable/readable by root.

The FLUENTD_DAEMON_USER / FLUENTD_DAEMON_GROUP variables are the only way to make this chart usable at the moment.

javsalgar commented 4 years ago

As reported here hostPath volumes are only writable/readable by root.

Well, the documentation only says writable. Theoretically you could make the logs folder readable by other users and it should work. But I will open a task for evaluating and implementing the option for the fluentd user to be root. I presume that it could be a value called fluentdUser which, by default it is set to fluentd (at least for now so it is not a major change in the chart). For your use cases it would be just setting that value to root. Would that make sense?

irizzant commented 4 years ago

Well, the documentation only says writable. Theoretically you could make the logs folder readable by other users and it should work

The documentation says writable, but as a matter of fact it's also readable only by root because:

But I will open a task for evaluating and implementing the option for the fluentd user to be root. I presume that it could be a value called fluentdUser which, by default it is set to fluentd (at least for now so it is not a major change in the chart). For your use cases it would be just setting that value to root

I don't think this would add any extra benefit to the chart, FLUENTD_DAEMON_USER / FLUENTD_DAEMON_GROUP variables already make the chart work as workaround. fluentdUser would just be a different option to set. What I'm suggesting is putting FLUENTD_DAEMON_USER / FLUENTD_DAEMON_GROUP as root as default to make the chart work with the default values, which is not at the moment

javsalgar commented 4 years ago

Hi.

I created this PR for discussion https://github.com/bitnami/charts/pull/2323

javsalgar commented 4 years ago

The issue was closed as the PR was merged. lease do not hesitate to re-open it if you find further issues.

mashail commented 4 years ago

Hey I am getting the same error on OCI OKE, using the helm chart version 1.2.0!

mashail commented 4 years ago

It turns out that an extra hostPath /u01/data has to be mounted as there is a symlink between the files under /var/lib and the files under /u01/data. So, mounting /u01/data too fixed the issue for us.

javsalgar commented 4 years ago

Hi,

Thanks for letting us know! :) It will be helpful for other OKE users.

xgbt commented 11 months ago

Setting FLUENT_UID to 0 didn't work for me. I had to add the following to forwarder.extraEnv

  - name: FLUENTD_DAEMON_USER
    value: root
  - name: FLUENTD_DAEMON_GROUP
    value: root

Since only the root user has access to /var/lib/docker/containers on the host.

It works, thanks..