Objects delivered in the wrong path in S3

max-blue commented 11 months ago

Describe the question/issue

I converted FluentD with Fluent-bit to ship logs from K8S to S3. The tag_rewrite config I have is not working as expected and pushing logs to the incorrect path in S3. Logs are expected to push in the following path:

2023/10/14/namespace/container_name/container-name-namespace_name-2023-10-14-UUID.txt

but it gets pushed in to the following path: 2023/10/14/var/log/containers/containers-var-20231014-0759-.log-object00N1PX3n

Configuration

fluentbitS3:
  enabled: true
  values:
    kind: DaemonSet
    image:
      repository: cr.fluentbit.io/fluent/fluent-bit
      pullPolicy: Always
    env:
      - name: CLUSTER
        value: company-cluster
    testFramework:
      enabled: true
      image:
        repository: busybox
        pullPolicy: Always
        tag: latest
    serviceAccount:
      create: true
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/company-cluster-fluentbit
    rbac:
      create: true
      nodeAccess: true
    hostNetwork: false
    dnsPolicy: ClusterFirst
    service:
      type: ClusterIP
      port: 2022
    livenessProbe:
      httpGet:
        path: /
        port: http
    readinessProbe:
      httpGet:
        path: /api/v1/health
        port: http
    flush: 1
    metricsPort: 2022

    config:
      service: |
        [SERVICE]
            Daemon Off
            Flush {{ .Values.flush }}
            Log_Level {{ .Values.logLevel }}
            Parsers_File parsers.conf
            Parsers_File custom_parsers.conf
            HTTP_Server On
            HTTP_Listen 0.0.0.0
            HTTP_Port {{ .Values.metricsPort }}
            Health_Check On

      inputs: |
        [INPUT]
            Name              tail
            Tag               s3logs.*
            Path              /var/log/containers/*.log
            parser            cri
            multiline.parser  cri         
            Mem_Buf_Limit     5MB
            Skip_Long_Lines   On
            Skip_Empty_Lines  On
            Refresh_Interval  10
        [FILTER]
            Name                kubernetes
            Match               s3logs.*
            Kube_Tag_Prefix     kube.var.log.containers.
            Merge_Log           On
            K8S-Logging.Parser  On
            K8S-Logging.Exclude On
            Keep_Log            Off
            Labels              Off   
            Annotations         Off         
        [FILTER]
            Name    record_modifier
            Match   s3logs.*
            Record  cluster_name ${CLUSTER}
        [FILTER]
            name    lua
            alias   set_std_keys
            match   s3logs.*
            script  /fluent-bit/scripts/s3_path.lua
            call    set_std_keys
        [FILTER]
            name rewrite_tag
            match s3logs.*
            rule $log ^.*$ s3.$namespace_name.$app_name.$container_name.$pod_id true               

      outputs: |
        [OUTPUT]
            Name s3
            Match s3logs.*
            bucket                       logs.company-cluster.us-east-1.company.com
            region                       us-east-1
            s3_key_format                /%Y/%m/%d/$TAG[1]/$TAG[2]/$TAG[3]/$TAG[3]-$TAG[1]-%Y%m%d-%H%M-${podid}-%{index}.%{file_extension}
            store_dir                    /var/log/fluentbit-s3-buffers
            total_file_size              256MB
            upload_timeout               2m
            use_put_object               On
            compression                  gzip
            preserve_data_ordering       On

    volumeMounts:
      - name: config
        mountPath: /fluent-bit/etc/fluent-bit.conf
        subPath: fluent-bit.conf
      - name: script
        mountPath: /fluent-bit/etc/s3_path.lua
        subPath: s3_path.lua
      - name: buffers
        mountPath: /var/log/fluentbit-s3-buffers

    daemonSetVolumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: buffers
        emptyDir: {} 
    daemonSetVolumeMounts:
      - name: varlog
        mountPath: /var/log
    logLevel: info
    luaScripts:
      s3_path.lua: |
        function set_std_keys(tag, timestamp, record)

            -- Pull up cluster
            if (record["cluster_name"] ~= nil) then
                record["cluster_name"] = record["cluster_name"]
            else
                record["cluster_name"] = "company-cluster"
            end

            if (record["kubernetes"] ~= nil) then
                kube = record["kubernetes"]

                -- Pull up namespace
                if (kube["namespace_name"] ~= nil and string.len(kube["namespace_name"]) > 0) then
                    record["namespace_name"] = kube["namespace_name"]
                else
                    record["namespace_name"] = "default"
                end

                -- Pull up container name
                if (kube["container_name"] ~= nil and string.len(kube["container_name"]) > 0) then
                    record["container_name"] = kube["container_name"]
                end

                -- Pull up pod id
                if (kube["pod_id"] ~= nil and string.len(kube["pod_id"]) > 0) then
                    record["pod_id"] = kube["pod_id"]
                end

                -- Pull up app name (Deployment, StateFuleSets, DaemonSet, Job, CronJob etc)
                if (kube["labels"] ~= nil) then
                    labels = kube["labels"]

                    if (labels["app"] ~= nil and string.len(labels["app"]) > 0) then
                        record["app_name"] = labels["app"]
                    elseif (labels["app.kubernetes.io/instance"] ~= nil and string.len(labels["app.kubernetes.io/instance"]) > 0) then
                        record["app_name"] = labels["app.kubernetes.io/instance"]
                    elseif (labels["k8s-app"] ~= nil and string.len(labels["k8s-app"]) > 0) then
                        record["app_name"] = labels["k8s-app"]
                    elseif (labels["name"] ~= nil and string.len(labels["name"]) > 0) then
                        record["app_name"] = labels["name"]
                    end
                else
                    record["app_name"] = record["app_name"]
                end
            end

          return 2, timestamp, record
        end

Fluent Bit Log Output

[2023/10/18 07:39:14] [debug] [input chunk] update output instances with new chunk size diff=346, records=1, input=emitter_for_rewrite_tag.3                                                                       │
│ [2023/10/18 07:39:14] [debug] [input chunk] update output instances with new chunk size diff=346, records=1, input=tail.0                                                                                          │
│ [2023/10/18 07:39:14] [debug] [input:tail:tail.0] inode=203429971, /var/log/containers/jumeirah-d647cd446-pm9z9_production_jumeirah-eb74b3c1f3a47ec7d5c922065122784847a5f31060917ae22633bdfca68907bd.log, events: IN │
│ _MODIFY                                                                                                                                                                                                            │
│ [2023/10/18 07:39:14] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information                                                                                                 │
│ [2023/10/18 07:39:15] [debug] [http_client] not using http_proxy for header                                                                                                                                        │
│ [2023/10/18 07:39:15] [debug] [http_client] server kubernetes.default.svc:443 will close connection #70                                                                                                            │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] Request (ns=production, pod=s.jumeirah-d647cd446-pm9z9) http_do=0, HTTP Status: 404                                                                │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] HTTP response                                                                                                                                       │
│ {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"s.jumeirah-d647cd446-pm9z9\" not found","reason":"NotFound","details":{"name":"s.jumeirah-d647cd446-pm9z9","kind":"pods"},"code":4 │
│ 04}                                                                                                                                                                                                                │
│                                                                                                                                                                                                                    │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=300, records=1, input=emitter_for_rewrite_tag.3                                                                       │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=329, records=1, input=emitter_for_rewrite_tag.3                                                                       │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=334, records=1, input=emitter_for_rewrite_tag.3                                                                       │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=963, records=3, input=tail.0                                                                                          │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information                                                                                                 │
│ [2023/10/18 07:39:15] [debug] [http_client] not using http_proxy for header                                                                                                                                        │
│ [2023/10/18 07:39:15] [debug] [http_client] server kubernetes.default.svc:443 will close connection #70                                                                                                            │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] Request (ns=production, pod=s.jumeirah-d647cd446-cvkvn) http_do=0, HTTP Status: 404                                                                │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] HTTP response                                                                                                                                       │
│ {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"s.jumeirah-d647cd446-cvkvn\" not found","reason":"NotFound","details":{"name":"s.jumeirah-d647cd446-cvkvn","kind":"pods"},"code":4 │
│ 04}                                                                                                                                                                                                                │
│                                                                                                                                                                                                                    │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=365, records=1, input=emitter_for_rewrite_tag.3                                                                       │
│ [2023/10/18 07:39:15] [debug] [input chunk] update output instances with new chunk size diff=365, records=1, input=tail.0                                                                                          │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information                                                                                                 │
│ [2023/10/18 07:39:15] [debug] [http_client] not using http_proxy for header                                                                                                                                        │
│ [2023/10/18 07:39:15] [debug] [http_client] server kubernetes.default.svc:443 will close connection #70                                                                                                            │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] Request (ns=production, pod=s.jumeirah-d647cd446-k424z) http_do=0, HTTP Status: 404                                                                │
│ [2023/10/18 07:39:15] [debug] [filter:kubernetes:kubernetes.0] HTTP response                                                                                                                                       │
│ {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"s.jumeirah-d647cd446-k424z\" not found","reason":"NotFound","details":{"name":"s.jumeirah-d647cd446-k424z","kind":"pods"},"code":4 │
│ 04}

Fluent Bit Version Info

Which AWS for Fluent Bit Versions have you tried? latest stable

Which versions have you seen the issue in? Are there any versions where you do not see the issue? All versions

Cluster Details

what is the networking setup? Cluster uses AWS CNI provided by AWS EKS
do you use App Mesh or a service mesh? NO
does you use VPC endpoints in a network restricted VPC? NO
EKS EC2 Daemon for Fluent Bit

Application Details

Unknown

Steps to reproduce issue

Use the configuration above and the logs go in the wrong path.

PettitWesley commented 11 months ago

The goal of this is to match all log messages right?

      $log ^.*$

Try this instead:

$log ^[\S]+$

I found this while working on this project, that was the regex that worked in FLB to match all logs- IIRC I tried the same regex you have and it didn't work, I am not sure why: https://github.com/aws/aws-for-fluent-bit/pull/499/files#diff-1413562a024b7a0a612040a520fe770ac13e9d3fcc799d78bd48a808a6230905R23

I'll add a debugging guide entry for this.

PettitWesley commented 11 months ago

I'll also update this tutorial as well: https://github.com/aws/aws-for-fluent-bit/tree/dev/use_cases/k8s-metadata-customize-tag

PettitWesley commented 11 months ago

https://github.com/aws/aws-for-fluent-bit/pull/749

max-blue commented 11 months ago

I am still not able to send logs to the correct path in s3. I have tried updated the config several different ways. Below is my latest config.

apiVersion: v1
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level debug
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2022
        Health_Check On

    [INPUT]
        Name              tail
        Tag               s3logs.*
        Path              /var/log/containers/*.log
        parser            cri
        multiline.parser  cri         
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Skip_Empty_Lines  On
        Refresh_Interval  10
    [FILTER]
        Name                kubernetes
        Match               s3logs.*
        Kube_Tag_Prefix     s3logs.var.log.containers.
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Keep_Log            Off
        Labels              Off   
        Annotations         Off      
    [FILTER]
        Name                rewrite_tag
        Match               s3logs.*
        Rule                $kubernetes['namespace_name']  ^[a-zA-Z0-9-_]*$ $kubernetes['namespace_name'].$kubernetes['container_name'].$kubernetes['pod_id']  false             
    [FILTER]
        Name    record_modifier
        Match   s3logs.*
        Record  cluster_name ${CLUSTER}            

    [OUTPUT]
        Name s3
        Match s3logs.*
        bucket                       logs.company.us-east-1.domain.com
        region                       us-east-1
        s3_key_format                /%Y/%m/%d/$TAG[1]/$TAG[2]/$TAG[2]-$TAG[1]-%Y%m%d-$TAG[3].txt
        store_dir                    /var/log/fluentbit-s3-buffers
        total_file_size              256MB
        upload_timeout               2m
        use_put_object               On
        compression                  gzip
        preserve_data_ordering       On

PettitWesley commented 11 months ago

Did you try my suggestion here? => https://github.com/aws/aws-for-fluent-bit/issues/748#issuecomment-1769725235

Also, your input sets this tag:

[INPUT]
Name tail Tag s3logs.*

But then the rewrite_tag rule will change the tag to start with $kubernetes['namespace_name']

But then your S3 match pattern is:

    Match s3logs.*

So your S3 output only matches logs which did not have their tag rewritten by the rewrite_tag filter I think .

aws / aws-for-fluent-bit