fluent-plugins-nursery / fluent-plugin-concat

Fluentd Filter plugin to concatenate multiline log separated in multiple events.
MIT License
108 stars 33 forks source link

Concatenating cri logs configError: Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required #110

Closed adityajalkhare closed 2 years ago

adityajalkhare commented 2 years ago

Problem

Background: I am trying to setup fluentd on Azure Kubernetes Service Cluster. It is running as a daemonset.

When using configuration provided in Readme.md for concatenating cri logs, pods are crashing with following error:

2021-11-05 09:25:35 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required"

...

Steps to replicate

Setup Fluentd using this yaml as base for azure blob as log store. Provided custom configuration through configmaps. Configuration is (tried with and without commented lines):

    <source>
      @type tail
      # @id in_tail_container_logs 
      path "#{ENV['FLUENT_CONTAINER_TAIL_PATH'] || '/var/log/pods/**/**/*.log'}" 
      # pos_file /var/log/fluentd-container.log.pos
      # exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
      # read_from_head true
      # rotate_wait 120
      <parse>
        @type regexp
        expression /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$/
        time_format "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT'] || '%Y-%m-%dT%H:%M:%S.%NZ'}"
      </parse>
      tag k8s
      @label @CONCAT
    </source>

    <label @CONCAT>
      <filter k8s>
        @type concat
        key message
        use_partial_cri_logtag true
        partial_cri_logtag_key logtag
        partial_cri_stream_key stream
      </filter>
      <match k8s>
        @type relabel
        @label @OUTPUT
      </match>
    </label>

    <label @OUTPUT>
      <match>
        @type azure-storage-append-blob
        @id out_azure_storage_append_blob

        azure_cloud                     "#{ENV['AZUREBLOB_AZURE_CLOUD']}"
        azure_storage_account           "#{ENV['AZUREBLOB_ACCOUNT_NAME']}"
        azure_storage_access_key        "#{ENV['AZUREBLOB_ACCOUNT_KEY']}"
        azure_storage_connection_string "#{ENV['AZUREBLOB_CONNECTION_STRING']}"
        azure_storage_sas_token         "#{ENV['AZUREBLOB_SAS_TOKEN']}"
        azure_container                 "#{ENV['AZUREBLOB_CONTAINER']}"
        azure_imds_api_version          "#{ENV['AZUREBLOB_IMDS_API_VERSION']}"
        azure_token_refresh_interval    "#{ENV['AZUREBLOB_TOKEN_REFRESH_INTERVAL']}"
        auto_create_container           true
        path                            "#{ENV['AZUREBLOB_LOG_PATH']}"
        azure_object_key_format         %{path}%{time_slice}_%{index}.log
        time_slice_format               %Y%m%d-%H
        # if you want to use %{tag} or %Y/%m/%d/ like syntax in path / azure_blob_name_format,
        # need to specify tag for %{tag} and time for %Y/%m/%d in <buffer> argument.
        <buffer>
          @type file
          path /var/log/fluent/azurestorageappendblob
          timekey 60 # 1 minute
          timekey_wait 60
          timekey_use_utc true # use utc
          chunk_limit_size 256m
        </buffer>
      </match>
    </label>

If I add use_partial_metadata true, then pods are comming in running state and logs are pushed to azure blob storage, but cri options are ignored and logs are not concatenated.

Expected Behavior

Pods must come in running state and logs are pushed to azure blob storage. ...

Your environment

LOCAL GEMS

addressable (2.8.0) azure-storage-blob (2.0.1) azure-storage-common (2.0.2) bigdecimal (default: 1.4.1) bundler (2.2.24, default: 1.17.2) cmath (default: 1.0.0) concurrent-ruby (1.1.9) connection_pool (2.2.5) cool.io (1.7.1) csv (default: 3.0.9) date (default: 2.0.0) dbm (default: 1.0.0) domain_name (0.5.20190701) e2mmap (default: 0.1.0) etc (default: 1.0.1) faraday (1.6.0) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) faraday_middleware (1.0.0) fcntl (default: 1.0.0) ffi (1.15.3) ffi-compiler (1.0.1) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-config-regexp-type (1.0.0) fluent-plugin-azure-storage-append-blob-lts (0.6.3) fluent-plugin-concat (2.4.0) fluent-plugin-detect-exceptions (0.0.13) fluent-plugin-grok-parser (2.6.2) fluent-plugin-json-in-json-2 (1.0.2) fluent-plugin-kubernetes_metadata_filter (2.7.2) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-parser-cri (0.1.1) fluent-plugin-prometheus (1.8.5) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-systemd (1.0.5) fluentd (1.13.3) forwardable (default: 1.2.0) gdbm (default: 2.0.0) http (4.4.1) http-accept (1.7.0) http-cookie (1.0.4) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.7.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) json (default: 2.1.0) jsonpath (1.1.0) kubeclient (4.9.2) logger (default: 1.3.0) lru_redux (1.1.0) matrix (default: 0.1.0) mime-types (3.3.1) mime-types-data (3.2021.0704) mini_portile2 (2.5.3) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) net-http-persistent (4.0.1) netrc (0.11.0) nokogiri (1.11.7 x86_64-linux) oj (3.11.0) openssl (default: 2.1.2) ostruct (default: 0.1.0) prime (default: 0.1.0) prometheus-client (0.9.0) psych (default: 3.1.0) public_suffix (4.0.6) quantile (0.2.1) racc (1.5.2) rake (13.0.6) rdoc (default: 6.1.2.1) recursive-open-struct (1.1.3) rest-client (2.1.0) rexml (default: 3.1.9.1) rss (default: 0.2.7) ruby2_keywords (0.0.5) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.5) strscan (default: 1.0.0) sync (default: 0.5.0) systemd-journal (1.4.2) thwait (default: 0.1.0) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.1) unf (0.1.4) unf_ext (0.0.7.7) webrick (1.7.0, default: 1.4.4) yajl-ruby (1.4.1) zlib (default: 1.0.0)

adityajalkhare commented 2 years ago

Gets resolved when using docker image - fluent/fluentd-kubernetes-daemonset:v1.14.2-debian-azureblob-1.0 Closing now.