permissions issue with system:anonymous

[cwiggs]

I'm getting the following error using kubectl logs -f fluentd-sumologic-b84rr

2019-04-02 16:37:47 +0000 [error]: config error file="/fluentd/etc/fluent.file.conf" error_class=Fluent::ConfigError error="start_namespace_watch: Exception encountered setting up namespace watch from Kubernetes API v1 endpoint https://10.100.0.1:443/api: namespaces is forbidden: User \"system:anonymous\" cannot list namespaces at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"namespaces is forbidden: User \\\"system:anonymous\\\" cannot list namespaces at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"namespaces\"},\"code\":403}\n)"

$ kubectl get pods/fluentd-sumologic-b84rr -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2019-04-02T16:36:54Z"
  generateName: fluentd-sumologic-
  labels:
    controller-revision-hash: "2277932056"
    name: fluentd-sumologic
    pod-template-generation: "1"
  name: fluentd-sumologic-b84rr
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: fluentd-sumologic
    uid: 86877936-5565-11e9-af43-0ec97014e648
  resourceVersion: "3665685"
  selfLink: /api/v1/namespaces/default/pods/fluentd-sumologic-b84rr
  uid: 868ac73c-5565-11e9-876a-0ae9293556dc
spec:
  containers:
  - env:
    - name: COLLECTOR_URL
      valueFrom:
        secretKeyRef:
          key: collector-url
          name: sumologic
    - name: VERIFY_SSL
      value: "false"
    - name: K8S_METADATA_FILTER_VERIFY_SSL
      value: "false"
    image: sumologic/fluentd-kubernetes-sumologic:latest
    imagePullPolicy: Always
    name: fluentd
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mnt/log/
      name: host-logs
      readOnly: true
    - mountPath: /var/log/
      name: host-logs
      readOnly: true
    - mountPath: /var/lib/docker/
      name: docker-logs
      readOnly: true
    - mountPath: /mnt/pos/
      name: pos-files
  dnsPolicy: ClusterFirst
  nodeName: ip-100-104-222-29.ec2.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: fluentd
  serviceAccountName: fluentd
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  volumes:
  - hostPath:
      path: /var/run/fluentd-pos
      type: ""
    name: pos-files
  - hostPath:
      path: /var/log/
      type: ""
    name: host-logs
  - hostPath:
      path: /var/lib/docker
      type: ""
    name: docker-logs
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-04-02T16:36:54Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-04-02T17:08:32Z"
    message: 'containers with unready status: [fluentd]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    message: 'containers with unready status: [fluentd]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-04-02T16:36:54Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://86a08c0cc3fa84e4fcfc437f3f87f770e812817cc14cb9ca885ee168ce583e5c
    image: sumologic/fluentd-kubernetes-sumologic:latest
    imageID: docker-pullable://sumologic/fluentd-kubernetes-sumologic@sha256:1bf2756525bb22ee3e4a1dce28f0adbfa11c3f9c2bd1202a58421a7fe89eb91b
    lastState:
      terminated:
        containerID: docker://86a08c0cc3fa84e4fcfc437f3f87f770e812817cc14cb9ca885ee168ce583e5c
        exitCode: 1
        finishedAt: "2019-04-02T17:08:31Z"
        reason: Error
        startedAt: "2019-04-02T17:08:30Z"
    name: fluentd
    ready: false
    restartCount: 11
    state:
      waiting:
        message: Back-off 5m0s restarting failed container=fluentd pod=fluentd-sumologic-b84rr_default(868ac73c-5565-11e9-876a-0ae9293556dc)
        reason: CrashLoopBackOff
  hostIP: <redacted>
  phase: Running
  podIP: <redacted>
  qosClass: BestEffort
  startTime: "2019-04-02T16:36:54Z"

It seems to me what is happening is fluentd isn't using the service account and defaulting to the anonymous user for some reason. I'm not sure how to look at /fluentd/etc/fluent.file.conf since the pod never actually starts so i'm not able to verify what user and/or service account it's using.

Any ideas what is causing the issue?

Thanks!

[cwiggs]

It seems like after a few hours the error goes away and fluentd starts to send some (I think some are missing) logs to sumologic, however I still can't find out what changed. destroying the daemon set and re-deploying it causes the issue to occur again.

[cosmo0920]

This repository isn't supported sumologic image and daemonset configuration.... Perhaps, do you use sumologic's kubernetes daemonset definition? https://github.com/SumoLogic/fluentd-kubernetes-sumologic/blob/master/daemonset/nonrbac/fluentd.yaml If yes, could you send your question into sumologic's k8s daemonset bug tracker instead of this repository?