Logs are not getting pushed to gcs bucket

[Vinayaks439]

image: fluent/fluentd:v1.12.0-debian-1.0 fluent-plugin-gcs version: "fluent-plugin-gcs", "0.4.0" Platform: Google Kubernetes Engine (GKE) Node Version: v1.17.14-gke.1600

Description: I'm running fluentd to ship logs to gcs bucket using the above mentioned image and plugin, but the issue that I'm facing is that the logs are not being shipped to my gcs bucket. And there are no exceptions in the logs of the pod either.

Config Map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: loki-grafana
data:
  fluent.conf: |-
    @include pods-fluent.conf
    @include loki-fluent.conf
    @include gcs-fluent.conf
  pods-fluent.conf: |-
    <source>
      @type tail
      read_from_head true
      tag kubernetes.*
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      exclude_path /var/log/containers/fluentd*
      path_key filename
      <parse>
        @type json
        time_type string
      </parse>
    </source>
    #########OPTIONAL################

    ######## IF you un-comment this you will get kuberenetes_meta data metrics in your logs#########
    # <filter kubernetes.**>
    #   @type kubernetes_metadata
    #   @id filter_kube_metadata
    #   kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
    #   verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
    #   ca_file "#{ENV['KUBERNETES_CA_FILE']}"
    #   skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
    #   skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
    #   skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
    #   skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
    #</filter>
  loki-fluent.conf: |-
    <match **>
        @type loki
        #### url of loki can be your loki-service inside the cluster or an external endpoint#############
        url "http://loki:3100"
        extra_labels {"env":"prod"}
        <label>
          filename
        </label>
        label_keys "app,job,namespace,pod"
        extract_kubernetes_labels true
        flush_interval 10s
        flush_at_shutdown true
        buffer_chuck_limit 1m
    </match>
  gcs-fluent.conf: |-
    <match **>
      @type gcs

      project <your-gcp-project>
      keyfile "#{ENV['GCP_CRED']}"
      bucket <BUCKET_NAME>
      object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
      path logs/${tag}/%Y/%m/%d/
      <buffer tag,time>
        @type file
        path /var/log/fluent/gcs
        timekey 1h
        timekey_wait 10min
        timekey_use_utc true
        flush_at_shutdown true
      </buffer>  
    </match>

Daemon Set file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: loki-grafana
  labels:
    app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      name: fluentd-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        name: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: <your-image>
        command: 
          - /bin/sh 
          - '-c'
          - >
            fluent-gem i fluent-plugin-grafana-loki-licence-fix ;
            tini /fluentd/entrypoint.sh;
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /fluentd/etc
        - name: google-cloud-key
          mountPath: /var/secrets/google
        env:
        - name: GCP_CRED
          value: /var/secrets/google/creds.json  
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: fluentd-config
      - name: google-cloud-key
        secret:
          secretName: fluentd-secret

Also seems like its not an auth issue as I was able to download a file from that bucket into my pod after install gcloud inside the pod.

[Vinayaks439]

I should have been using copy function to ship it both gcs and loki at the same time.

FYR: https://docs.fluentd.org/output/copy