Error reading logs in k8s

fluentd or td-agent version. 1.2.0
Environment information, e.g. OS.

K8S version 1.10.0

Your configuration

trying to deploy using a deamonset in pre baked docker image: my docker file :

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb

FROM debian:stretch-slim
LABEL maintainer "TAGOMORI Satoshi <tagomoris@gmail.com>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.1"

ENV DUMB_INIT_VERSION=1.2.0

ENV GOSU_VERSION=1.10

ARG DEBIAN_FRONTEND=noninteractive

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update \
 && apt-get upgrade -y \
 && apt-get install -y --no-install-recommends \
            ca-certificates \
            ruby \
 && buildDeps=" \
      make gcc g++ libc-dev \
      ruby-dev \
      wget bzip2 gnupg dirmngr \
    " \
 && apt-get install -y --no-install-recommends $buildDeps \
 && update-ca-certificates \
 && echo 'gem: --no-document' >> /etc/gemrc \
 && gem install oj -v 3.3.10 \
 && gem install json -v 2.1.0 \
 && gem install fluentd -v 1.2.0 \
 && gem install fluent-plugin-redis \
 && dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
 && wget -O /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_$dpkgArch \
 && chmod +x /usr/bin/dumb-init \
 && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
 && wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
 && export GNUPGHOME="$(mktemp -d)" \
 && gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc \
 && chmod +x /usr/local/bin/gosu \
 && gosu nobody true \
 && wget -O /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
 && cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
 && ./configure && make \
 && mv lib/libjemalloc.so.2 /usr/lib \
 && apt-get purge -y --auto-remove \
"Dockerfile" 77L, 2901C# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb

FROM debian:stretch-slim
LABEL maintainer "TAGOMORI Satoshi <tagomoris@gmail.com>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.1"

ENV DUMB_INIT_VERSION=1.2.0

ENV GOSU_VERSION=1.10

ARG DEBIAN_FRONTEND=noninteractive

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update \
 && apt-get upgrade -y \
 && apt-get install -y --no-install-recommends \
            ca-certificates \
            ruby \
 && buildDeps=" \
      make gcc g++ libc-dev \
      ruby-dev \
      wget bzip2 gnupg dirmngr \
    " \
 && apt-get install -y --no-install-recommends $buildDeps \
 && update-ca-certificates \
 && echo 'gem: --no-document' >> /etc/gemrc \
 && gem install oj -v 3.3.10 \
 && gem install json -v 2.1.0 \
 && gem install fluentd -v 1.2.0 \
 && gem install fluent-plugin-redis \
 && dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
 && wget -O /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_$dpkgArch \
 && chmod +x /usr/bin/dumb-init \
 && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
 && wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
 && export GNUPGHOME="$(mktemp -d)" \
 && gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc \
 && chmod +x /usr/local/bin/gosu \
&& gosu nobody true \
 && wget -O /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
 && cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
 && ./configure && make \
 && mv lib/libjemalloc.so.2 /usr/lib \
 && apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

# for log storage (maybe shared with host)
RUN mkdir -p /fluentd/log
# configuration/plugins path (default: copied from .)
RUN mkdir -p /fluentd/etc /fluentd/plugins

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/
RUN chmod +x /bin/entrypoint.sh

ONBUILD COPY fluent.conf /fluentd/etc/
ONBUILD COPY plugins /fluentd/plugins/

ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
ENV DUMB_INIT_SETSID 0

EXPOSE 24224 5140

ENTRYPOINT ["/bin/entrypoint.sh"]

CMD exec fluentd -c /fluentd/etc/${FLUENTD_CONF} -p /fluentd/plugins $FLUENTD_OPT

my entrypoint.sh:

#!/usr/bin/dumb-init /bin/sh

uid=${FLUENT_UID:-1000}

# check if a old fluent user exists and delete it
cat /etc/passwd | grep fluent
if [ $? -eq 0 ]; then
    deluser fluent
fi

# (re)add the fluent user with $FLUENT_UID
useradd -u ${uid} -o -c "" -m fluent
export HOME=/home/fluent

# chown home and data folder
chown -R fluent /home/fluent
chown -R fluent /fluentd

exec gosu fluent "$@"

my k8s Deamonset yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-logging
  namespace: monitoring
  labels:
    name: fluentd-logging
    version: v1.2.1
spec:
  selector:
    matchLabels:
      name: fluentd-logging
  template:
    metadata:
      labels:
        name: fluentd-logging
        version: v1.2.1
    spec:
      containers:
      - name: fluentd
        image: 085551151339.dkr.ecr.eu-west-1.amazonaws.com/fluentd:app
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluentd-config
          mountPath: /fluentd/etc/
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentd-config
        configMap:
          name: fluentd-config
      serviceAccountName: fluentd-logging

my config file (fluent.conf) :

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config
  namespace: monitoring
  labels:
    k8s-app: fluentd-config
data:
  fluent.conf: |
    @include application.conf
    <match **>
      @type redis

      host logstash-redis.gtforge.com
      port 6379
      insert_key_prefix fluentd_prod

      # database number is optional.
      # db_number 0        # 0 is default
      # If requirepass is set, please specify this.
      # password hogefuga
      # Users can set '${tag}' or ${tag[0]}.${tag[1]} or ...?
      # insert_key_prefix '${tag}'
      # Users can set strftime format.
      # "%s" will be replaced into unixtime.
      # "%Y%m%d.%H%M%S" will be replaced like as 20161202.112335.
      # strftime_format "%s"
      # Allow insert key duplicate. It will work as update values.
      # allow_duplicate_key true
      # ttl 300 # If 0 or negative value is set, ttl is not set in each key.
    </match>

  application.conf: |

    <match kubernetes.var.log.containers.**fluentd**.log>
      @type null
    </match>

    <source>
      @type tail
      path /var/log/containers/*.log
      tag kube.*
      read_from_head true
      format json
       <parse>
         @type none
       </parse>
    </source>

after deploying to kubernetes pods are coming up and recieving the following error (for example):

2018-05-03 13:47:31 +0000 [warn]: #0 /var/log/containers/fluentd-logging-46phg_monitoring_fluentd-33e87e34dae2ff6bc3e98c55df9730171e756d2d9e91355b96d169838e4c2870.log unreadable. It is excluded and would be examined next time. I've tried to play with a lot, can't find any solution...

Please advise.

fluent / fluentd-docker-image

Error reading logs in k8s #116