Concatenating cri logs configError: Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required

Problem

Background: I am trying to setup fluentd on Azure Kubernetes Service Cluster. It is running as a daemonset.

Kubernetes Version: 1.20.5
Fluentd-Kubernetes-daemonset docker image: fluent/fluentd-kubernetes-daemonset:v1.13.3-debian-azureblob-1.2

When using configuration provided in Readme.md for concatenating cri logs, pods are crashing with following error:

2021-11-05 09:25:35 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required"

...

Steps to replicate

Setup Fluentd using this yaml as base for azure blob as log store. Provided custom configuration through configmaps. Configuration is (tried with and without commented lines):

    <source>
      @type tail
      # @id in_tail_container_logs 
      path "#{ENV['FLUENT_CONTAINER_TAIL_PATH'] || '/var/log/pods/**/**/*.log'}" 
      # pos_file /var/log/fluentd-container.log.pos
      # exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
      # read_from_head true
      # rotate_wait 120
      <parse>
        @type regexp
        expression /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$/
        time_format "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT'] || '%Y-%m-%dT%H:%M:%S.%NZ'}"
      </parse>
      tag k8s
      @label @CONCAT
    </source>

    <label @CONCAT>
      <filter k8s>
        @type concat
        key message
        use_partial_cri_logtag true
        partial_cri_logtag_key logtag
        partial_cri_stream_key stream
      </filter>
      <match k8s>
        @type relabel
        @label @OUTPUT
      </match>
    </label>

    <label @OUTPUT>
      <match>
        @type azure-storage-append-blob
        @id out_azure_storage_append_blob

        azure_cloud                     "#{ENV['AZUREBLOB_AZURE_CLOUD']}"
        azure_storage_account           "#{ENV['AZUREBLOB_ACCOUNT_NAME']}"
        azure_storage_access_key        "#{ENV['AZUREBLOB_ACCOUNT_KEY']}"
        azure_storage_connection_string "#{ENV['AZUREBLOB_CONNECTION_STRING']}"
        azure_storage_sas_token         "#{ENV['AZUREBLOB_SAS_TOKEN']}"
        azure_container                 "#{ENV['AZUREBLOB_CONTAINER']}"
        azure_imds_api_version          "#{ENV['AZUREBLOB_IMDS_API_VERSION']}"
        azure_token_refresh_interval    "#{ENV['AZUREBLOB_TOKEN_REFRESH_INTERVAL']}"
        auto_create_container           true
        path                            "#{ENV['AZUREBLOB_LOG_PATH']}"
        azure_object_key_format         %{path}%{time_slice}_%{index}.log
        time_slice_format               %Y%m%d-%H
        # if you want to use %{tag} or %Y/%m/%d/ like syntax in path / azure_blob_name_format,
        # need to specify tag for %{tag} and time for %Y/%m/%d in <buffer> argument.
        <buffer>
          @type file
          path /var/log/fluent/azurestorageappendblob
          timekey 60 # 1 minute
          timekey_wait 60
          timekey_use_utc true # use utc
          chunk_limit_size 256m
        </buffer>
      </match>
    </label>

If I add use_partial_metadata true, then pods are comming in running state and logs are pushed to azure blob storage, but cri options are ignored and logs are not concatenated.

Expected Behavior

Pods must come in running state and logs are pushed to azure blob storage. ...

Your environment

OS version = Ubuntu 18.04.5 LTS
Kernel = 5.4.0-1062-azure
CONTAINER-RUNTIME = containerd://1.4.4+azure
Platform = AKS - Kubernetes - 1.20.5
Fluentd Version = fluentd 1.13.3

plugin version =

2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob-lts' version '0.6.3'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.7.2'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.5'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-11-05 09:25:35 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-11-05 09:25:35 +0000 [info]: gem 'fluentd' version '1.13.3'

fluent-gem list:

LOCAL GEMS

addressable (2.8.0) azure-storage-blob (2.0.1) azure-storage-common (2.0.2) bigdecimal (default: 1.4.1) bundler (2.2.24, default: 1.17.2) cmath (default: 1.0.0) concurrent-ruby (1.1.9) connection_pool (2.2.5) cool.io (1.7.1) csv (default: 3.0.9) date (default: 2.0.0) dbm (default: 1.0.0) domain_name (0.5.20190701) e2mmap (default: 0.1.0) etc (default: 1.0.1) faraday (1.6.0) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) faraday_middleware (1.0.0) fcntl (default: 1.0.0) ffi (1.15.3) ffi-compiler (1.0.1) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-config-regexp-type (1.0.0) fluent-plugin-azure-storage-append-blob-lts (0.6.3) fluent-plugin-concat (2.4.0) fluent-plugin-detect-exceptions (0.0.13) fluent-plugin-grok-parser (2.6.2) fluent-plugin-json-in-json-2 (1.0.2) fluent-plugin-kubernetes_metadata_filter (2.7.2) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-parser-cri (0.1.1) fluent-plugin-prometheus (1.8.5) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-systemd (1.0.5) fluentd (1.13.3) forwardable (default: 1.2.0) gdbm (default: 2.0.0) http (4.4.1) http-accept (1.7.0) http-cookie (1.0.4) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.7.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) json (default: 2.1.0) jsonpath (1.1.0) kubeclient (4.9.2) logger (default: 1.3.0) lru_redux (1.1.0) matrix (default: 0.1.0) mime-types (3.3.1) mime-types-data (3.2021.0704) mini_portile2 (2.5.3) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) net-http-persistent (4.0.1) netrc (0.11.0) nokogiri (1.11.7 x86_64-linux) oj (3.11.0) openssl (default: 2.1.2) ostruct (default: 0.1.0) prime (default: 0.1.0) prometheus-client (0.9.0) psych (default: 3.1.0) public_suffix (4.0.6) quantile (0.2.1) racc (1.5.2) rake (13.0.6) rdoc (default: 6.1.2.1) recursive-open-struct (1.1.3) rest-client (2.1.0) rexml (default: 3.1.9.1) rss (default: 0.2.7) ruby2_keywords (0.0.5) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.5) strscan (default: 1.0.0) sync (default: 0.5.0) systemd-journal (1.4.2) thwait (default: 0.1.0) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.1) unf (0.1.4) unf_ext (0.0.7.7) webrick (1.7.0, default: 1.4.4) yajl-ruby (1.4.1) zlib (default: 1.0.0)

fluent-plugins-nursery / fluent-plugin-concat