cri-logs + traefik issues, not passing cri parsed values?

I leverage containerd + traefik in my system. However I am getting 0 parsed traefik logs for some reason. I output them as JSON.

From the logs, it appears cri-logs does run and process things, then traefik parser kicks in but is operating on the CRI entry instead of the extracted JSON.

time="31-08-2023 11:57:33" level=debug msg="pushing {Raw:2023-08-31T17:57:33.578239803Z stdout F {\"ClientAddr\":\"10.42.0.111:47264\",\"ClientHost\":\"10.42.0.111\",\"ClientPort\":\"47264\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":58,\"DownstreamStatus\":200,\"Duration\":6142285,\"OriginContentSize\":58,\"OriginDuration\":5951640,\"OriginStatus\":200,\"Overhead\":190645,\"RequestAddr\":\"foo.test.com\",\"RequestContentSize\":0,\"RequestCount\":2300,\"RequestHost\":\"foo.test.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/foo/test\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"foo-test-router@kubernetes\",\"ServiceAddr\":\"10.42.5.114:4000\",\"ServiceName\":\"kube-system-blocky-4000@kubernetes\",\"ServiceURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.42.5.114:4000\",\"Path\":\"\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"StartLocal\":\"2023-08-31T17:57:33.569513173Z\",\"StartUTC\":\"2023-08-31T17:57:33.569513173Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"downstream_Content-Length\":\"58\",\"downstream_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"downstream_Content-Type\":\"application/json\",\"downstream_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"downstream_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"downstream_Vary\":\"Origin\",\"downstream_X-Content-Type-Options\":\"nosniff\",\"downstream_X-Frame-Options\":\"DENY\",\"downstream_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"downstream_X-Xss-Protection\":\"1\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"origin_Content-Length\":\"58\",\"origin_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"origin_Content-Type\":\"application/json\",\"origin_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"origin_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"origin_Vary\":\"Origin\",\"origin_X-Content-Type-Options\":\"nosniff\",\"origin_X-Frame-Options\":\"DENY\",\"origin_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"origin_X-Xss-Protection\":\"1\",\"request_Accept\":\"*/*\",\"request_Accept-Encoding\":\"gzip, deflate, br\",\"request_User-Agent\":\"foo/2023.8.4 httpx/0.24.1 Python/3.11\",\"request_X-Forwarded-Host\":\"foo.test.com\",\"request_X-Forwarded-Port\":\"443\",\"request_X-Forwarded-Proto\":\"https\",\"request_X-Forwarded-Server\":\"traefik-7c47777bf-jdsgk\",\"request_X-Geo-City\":\"-\",\"request_X-Geo-Countrylong\":\"-\",\"request_X-Geo-Countryshort\":\"-\",\"request_X-Geo-Latitude\":\"0\",\"request_X-Geo-Longitude\":\"0\",\"request_X-Geo-Region\":\"-\",\"request_X-Geo-Timezone\":\"-\",\"request_X-Geo-Zipcode\":\"-\",\"request_X-Real-Ip\":\"10.42.0.111\",\"time\":\"2023-08-31T17:57:33Z\"} Src:/var/log/containers/traefik-7c47777bf-jdsgk_kube-system_traefik-9ef9547d3d74c6cb6dfc7eb30d4a0da00f80503c70c822b167c467457f32490f.log Time:2023-08-31 11:57:33.579180676 -0600 MDT m=+280.328039738 Labels:map[program:traefik type:containerd] Process:true Module:file}" tail=/var/log/containers/traefik-7c47777bf-jdsgk_kube-system_traefik-9ef9547d3d74c6cb6dfc7eb30d4a0da00f80503c70c822b167c467457f32490f.log type=file

time="31-08-2023 11:57:33" level=debug msg="+ Grok '^%{TI...' returned 4 entries to merge in Parsed" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="\t.Parsed['cri_timestamp'] = '2023-08-31T17:57:33.578239803Z'" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="\t.Parsed['stream'] = 'stdout'" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="\t.Parsed['logtag'] = 'F'" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="\t.Parsed['message'] = '{\"ClientAddr\":\"10.42.0.111:47264\",\"ClientHost\":\"10.42.0.111\",\"ClientPort\":\"47264\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":58,\"DownstreamStatus\":200,\"Duration\":6142285,\"OriginContentSize\":58,\"OriginDuration\":5951640,\"OriginStatus\":200,\"Overhead\":190645,\"RequestAddr\":\"foo.test.com\",\"RequestContentSize\":0,\"RequestCount\":2300,\"RequestHost\":\"foo.test.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/foo/test\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"foo-test-router@kubernetes\",\"ServiceAddr\":\"10.42.5.114:4000\",\"ServiceName\":\"kube-system-blocky-4000@kubernetes\",\"ServiceURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.42.5.114:4000\",\"Path\":\"\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"StartLocal\":\"2023-08-31T17:57:33.569513173Z\",\"StartUTC\":\"2023-08-31T17:57:33.569513173Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"downstream_Content-Length\":\"58\",\"downstream_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"downstream_Content-Type\":\"application/json\",\"downstream_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"downstream_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"downstream_Vary\":\"Origin\",\"downstream_X-Content-Type-Options\":\"nosniff\",\"downstream_X-Frame-Options\":\"DENY\",\"downstream_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"downstream_X-Xss-Protection\":\"1\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"origin_Content-Length\":\"58\",\"origin_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"origin_Content-Type\":\"application/json\",\"origin_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"origin_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"origin_Vary\":\"Origin\",\"origin_X-Content-Type-Options\":\"nosniff\",\"origin_X-Frame-Options\":\"DENY\",\"origin_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"origin_X-Xss-Protection\":\"1\",\"request_Accept\":\"*/*\",\"request_Accept-Encoding\":\"gzip, deflate, br\",\"request_User-Agent\":\"foo/2023.8.4 httpx/0.24.1 Python/3.11\",\"request_X-Forwarded-Host\":\"foo.test.com\",\"request_X-Forwarded-Port\":\"443\",\"request_X-Forwarded-Proto\":\"https\",\"request_X-Forwarded-Server\":\"traefik-7c47777bf-jdsgk\",\"request_X-Geo-City\":\"-\",\"request_X-Geo-Countrylong\":\"-\",\"request_X-Geo-Countryshort\":\"-\",\"request_X-Geo-Latitude\":\"0\",\"request_X-Geo-Longitude\":\"0\",\"request_X-Geo-Region\":\"-\",\"request_X-Geo-Timezone\":\"-\",\"request_X-Geo-Zipcode\":\"-\",\"request_X-Real-Ip\":\"10.42.0.111\",\"time\":\"2023-08-31T17:57:33Z\"}'" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ok" id=rough-voice name=child-crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="child is success, OnSuccess=next_stage, skip" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="+ Processing 5 statics" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg=".Parsed[logsource] = 'cri'" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="setting target StrTime to 2023-08-31T17:57:33.578239803Z"

time="31-08-2023 11:57:33" level=debug msg="evt.StrTime = '2023-08-31T17:57:33.578239803Z'" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg=".Parsed[program] = 'traefik'" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg=".Meta[datasource_path] = '/var/log/containers/traefik-7c47777bf-jdsgk_kube-system_traefik-9ef9547d3d74c6cb6dfc7eb30d4a0da00f80503c70c822b167c467457f32490f.log'" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg=".Meta[datasource_type] = 'file'" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ok" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="move Event from stage s00-raw to s01-parse" id=aged-bush name=crowdsecurity/cri-logs stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="node successful, stop end stage s00-raw" node-name=aged-bush stage=s00-raw

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko (failed filter)" id=quiet-water name=crowdsecurity/dropbear-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko (failed filter)" id=little-moon name=crowdsecurity/iptables-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko (failed filter)" id=wandering-butterfly name=crowdsecurity/mysql-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko (failed filter)" id=falling-smoke name=crowdsecurity/sshd-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="+ Grok '%{NGI...' didn't return data on '{\"ClientAddr\":\"10.42.0.111:47264\",\"ClientHost\":\"10.42.0.111\",\"ClientPort\":\"47264\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":58,\"DownstreamStatus\":200,\"Duration\":6142285,\"OriginContentSize\":58,\"OriginDuration\":5951640,\"OriginStatus\":200,\"Overhead\":190645,\"RequestAddr\":\"foo.test.com\",\"RequestContentSize\":0,\"RequestCount\":2300,\"RequestHost\":\"foo.test.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/foo/test\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"foo-test-router@kubernetes\",\"ServiceAddr\":\"10.42.5.114:4000\",\"ServiceName\":\"kube-system-blocky-4000@kubernetes\",\"ServiceURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.42.5.114:4000\",\"Path\":\"\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"StartLocal\":\"2023-08-31T17:57:33.569513173Z\",\"StartUTC\":\"2023-08-31T17:57:33.569513173Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"downstream_Content-Length\":\"58\",\"downstream_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"downstream_Content-Type\":\"application/json\",\"downstream_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"downstream_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"downstream_Vary\":\"Origin\",\"downstream_X-Content-Type-Options\":\"nosniff\",\"downstream_X-Frame-Options\":\"DENY\",\"downstream_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"downstream_X-Xss-Protection\":\"1\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"origin_Content-Length\":\"58\",\"origin_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"origin_Content-Type\":\"application/json\",\"origin_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"origin_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"origin_Vary\":\"Origin\",\"origin_X-Content-Type-Options\":\"nosniff\",\"origin_X-Frame-Options\":\"DENY\",\"origin_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"origin_X-Xss-Protection\":\"1\",\"request_Accept\":\"*/*\",\"request_Accept-Encoding\":\"gzip, deflate, br\",\"request_User-Agent\":\"foo/2023.8.4 httpx/0.24.1 Python/3.11\",\"request_X-Forwarded-Host\":\"foo.test.com\",\"request_X-Forwarded-Port\":\"443\",\"request_X-Forwarded-Proto\":\"https\",\"request_X-Forwarded-Server\":\"traefik-7c47777bf-jdsgk\",\"request_X-Geo-City\":\"-\",\"request_X-Geo-Countrylong\":\"-\",\"request_X-Geo-Countryshort\":\"-\",\"request_X-Geo-Latitude\":\"0\",\"request_X-Geo-Longitude\":\"0\",\"request_X-Geo-Region\":\"-\",\"request_X-Geo-Timezone\":\"-\",\"request_X-Geo-Zipcode\":\"-\",\"request_X-Real-Ip\":\"10.42.0.111\",\"time\":\"2023-08-31T17:57:33Z\"}'" id=rough-brook name=child-crowdsecurity/traefik-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko" id=rough-brook name=child-crowdsecurity/traefik-logs stage=s01-parse

time="31-08-2023 11:57:33" level=error msg="UnmarshalJSON : invalid character '-' after top-level value" line="2023-08-31T17:57:33.578239803Z stdout F {\"ClientAddr\":\"10.42.0.111:47264\",\"ClientHost\":\"10.42.0.111\",\"ClientPort\":\"47264\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":58,\"DownstreamStatus\":200,\"Duration\":6142285,\"OriginContentSize\":58,\"OriginDuration\":5951640,\"OriginStatus\":200,\"Overhead\":190645,\"RequestAddr\":\"foo.test.com\",\"RequestContentSize\":0,\"RequestCount\":2300,\"RequestHost\":\"foo.test.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/foo/test\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"foo-test-router@kubernetes\",\"ServiceAddr\":\"10.42.5.114:4000\",\"ServiceName\":\"kube-system-blocky-4000@kubernetes\",\"ServiceURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.42.5.114:4000\",\"Path\":\"\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"StartLocal\":\"2023-08-31T17:57:33.569513173Z\",\"StartUTC\":\"2023-08-31T17:57:33.569513173Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"downstream_Content-Length\":\"58\",\"downstream_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"downstream_Content-Type\":\"application/json\",\"downstream_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"downstream_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"downstream_Vary\":\"Origin\",\"downstream_X-Content-Type-Options\":\"nosniff\",\"downstream_X-Frame-Options\":\"DENY\",\"downstream_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"downstream_X-Xss-Protection\":\"1\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"origin_Content-Length\":\"58\",\"origin_Content-Security-Policy\":\"frame-ancestors 'self' *.foo.test.com\",\"origin_Content-Type\":\"application/json\",\"origin_Date\":\"Thu, 31 Aug 2023 17:57:33 GMT\",\"origin_Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"origin_Vary\":\"Origin\",\"origin_X-Content-Type-Options\":\"nosniff\",\"origin_X-Frame-Options\":\"DENY\",\"origin_X-Robots-Tag\":\"none,noarchive,nosnippet,notranslate,noimageindex,\",\"origin_X-Xss-Protection\":\"1\",\"request_Accept\":\"*/*\",\"request_Accept-Encoding\":\"gzip, deflate, br\",\"request_User-Agent\":\"foo/2023.8.4 httpx/0.24.1 Python/3.11\",\"request_X-Forwarded-Host\":\"foo.test.com\",\"request_X-Forwarded-Port\":\"443\",\"request_X-Forwarded-Proto\":\"https\",\"request_X-Forwarded-Server\":\"traefik-7c47777bf-jdsgk\",\"request_X-Geo-City\":\"-\",\"request_X-Geo-Countrylong\":\"-\",\"request_X-Geo-Countryshort\":\"-\",\"request_X-Geo-Latitude\":\"0\",\"request_X-Geo-Longitude\":\"0\",\"request_X-Geo-Region\":\"-\",\"request_X-Geo-Timezone\":\"-\",\"request_X-Geo-Zipcode\":\"-\",\"request_X-Real-Ip\":\"10.42.0.111\",\"time\":\"2023-08-31T17:57:33Z\"}"

time="31-08-2023 11:57:33" level=warning msg="failed to run filter : invalid character '-' after top-level value (1:1)\n | UnmarshalJSON(evt.Line.Raw, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-tree name=child-crowdsecurity/traefik-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko" id=restless-tree name=child-crowdsecurity/traefik-logs stage=s01-parse

time="31-08-2023 11:57:33" level=debug msg="Event leaving node : ko" id=snowy-pine name=crowdsecurity/traefik-logs stage=s01-parse

I am wondering why it appears that CRI is successful, then traefik starts operating on the same line instead of the parsed line?

e.g., UnmarshalJSON : invalid character '-' after top-level value" line="2023-08-31T17:57:33.578239803Z stdout F {\"ClientAddr\":\"10.42.0.111:47....rest of json -- Note it prefixed with CRI formatting.

I could also be misunderstanding.

Thanks.

Subset of my helm arguments.

container_runtime: containerd
# ...
acquisition:
    # The namespace where the pod is located
    - namespace: kube-system
      podName: traefik-*
      program: traefik
# ...
env:
  - name: PARSERS
    value: >-
      crowdsecurity/cri-logs
      crowdsecurity/whitelists
      crowdsecurity/sshd-logs
  - name: COLLECTIONS
    value: >-
      crowdsecurity/linux
      crowdsecurity/sshd
      crowdsecurity/traefik
      crowdsecurity/base-http-scenarios
      crowdsecurity/http-cve
      crowdsecurity/whitelist-good-actors
      crowdsecurity/mysql
  - name: LEVEL_DEBUG
    value: "true"
# ....

crowdsecurity / hub

cri-logs + traefik issues, not passing cri parsed values? #816