Permission problem access /var/log/containers/*

[jfillman]

This issue must have come up before so apologies for not finding it.

My fluentd containers are now run as user '1000' and so don't have read access to /var/lib/docker/containers which is owned by root and has 0700 perms.

I get the log error mesg:

unreadable. It is excluded and would be examined next time. I can't seem to figure out how to run my container as root or grant the necessary perms on the volume. Adding a securitycontext didn't work I'm a kubernetes beginner so hand holding is much appreciated. Here's my daemonset: ``` apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule securityContext: runAsNonRoot: false runAsUser: 0 containers: - name: fluentd image: /fluentd-syslog-ng securityContext: runAsNonRoot: false runAsUser: 0 imagePullPolicy: Always env: - name: SYSLOG_SERVER value: "SYSLOGSERVER" - name: SYSLOG_PORT value: "516" - name: API_SERVER value: "https://FQDN" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: vartmp mountPath: /var/tmp - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: vartmp hostPath: path: /var/tmp - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers ```

[flowstate]

This is also happening to me. I can run bash on the box as the same user (root) and cat all the logs. But I am also getting the 'log unreadable' error. Any chance we can get some help? It's really disappointing that there's been no feedback on this issue for 28 days.

[pirankashani]

Hi, I had same problem and solved when I delete --lod-driver=journald in docker.conf file. If you check the folder /var/log/containers it would be empty and that's why you receive this error message.

[mvernimmen]

Could you try this?

securityContext:
   readOnlyRootFilesystem: true

[edsiper]

Considering this ticket as solved due to previous answers.

[jiuchongxiao]

@jfillman did you solved it? and how solved? i have the same issue.

[xinghui-hou]

I have same problem . flunt-bit output log

[2019/10/11 10:43:59] [error] [in_tail] read error, check permissions: /home/docker/containers/*/*.log check the directory permissions 0755 is ok

[gotoworld]

same issue

[2020/02/26 03:53:13] [error] [in_tail] read error, check permissions: /data/ratelimit-logs/gateway_2.log [2020/02/26 03:53:13] [error] [in_tail] read error, check permissions: /data/ratelimit-logs/gateway_ratelimit.log

[repeatedly]

This is not fluent-bit daemonset repository. If you have a problem with fluent-bit, post it to correct place.

[jpiper]

I had this problem and it was due to SELinux profiles

[perezmark]

@jpiper how did you solve it?

[jpiper]

I just set SELinux to permissive. Not really a solution, just a workaround

[nhenderson]

@jpiper and @perezmark

Updating the containers securityContext and adding the spc_t SELinux policy to the fluentd DaemonSet worked for me:

Like so:

containers:
- name: fluentd
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
    seLinuxOptions:
      type: spc_t

This may be a better solution than just turning off SELinux on the all the nodes.