the docker log indexed as type keyword

[dengqinghua]

Problem

The field log is indexed as keyword.

The log field is the docker container's log, and I checked the map, the field type is keyword, instead of text

{
  "mappings": {
    "_doc": {
      "dynamic_templates": [
        {
          "message_full": {
            "match": "message_full",
            "mapping": {
              "fields": {
                "keyword": {
                 ...
                  "type": "keyword"
                }
              },
              "type": "text"
            }
          }
        },
        {
          "message": {
            "match": "message",
            "mapping": {
              "type": "text"
            }
          }
        }
....
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
       ...omited...
        # this filed is setted as keyword
        "log": {
          "type": "keyword"
        },
        "stream": {
          "type": "keyword"
      ...
  }
}

Steps to replicate

I used the fluentd-kubernetes-daemonset the config

Here is the daemonSet of my config in k8s

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-elasticsearch7-1.1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "---"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENT_ELASTICSEARCH_USER
            value: "elastic"
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            value: "---"
          - name: FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT
            value: "%Y.%m"
          - name: FLUENT_UID
            value: "0"

        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Expected Behavior or What you need to ask

The docker container's log field should be index as type text.

Using Fluentd and ES plugin versions

• OS version: Centos 7 x86_64 x86_64 x86_64 GNU/Linux
• Bare Metal or within Docker or Kubernetes or others? : Kubernetes
• Fluentd v0.12 or v0.14/v1.0
  • fluent/fluentd:v1.15.1-debian-amd64-1.0
• ES version 7.10.1