What happened:
Updated some regex to match the timestamp and upgraded the SCK. However, when the helm was upgraded with a new value file few pods went to Crashloopbackoff.
I reverted the changes and upgraded the helm chart but the problem persisted. I tried to uninstall and install the helm chart again but the issue remains there.
Prior to this change, the logs were being sent from some of the pods. However, multiline was not working so tried to fix the regex.
The following error logs can be seen in pod logs:
bundler: failed to load command: fluentd (/usr/bin/fluentd)
/usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:92:in parse_error!': unexpected end of file in a single quoted string at output.conf line 267,138 (Fluent::ConfigParseError) 266: @type jq_transformer 267: jq '.record | . + (.source | capture("/var/log/pods/(?<pod_uid>[^/]+)/(?<container_name>[^/]+)/(?<container_retry>[0-9]+).log")) ------------------------------------------------------------------------------------------------------------------------------------------^ 268: </filter> from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:131:inscan_single_quoted_string'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:84:in scan_string' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:75:inparse_literal'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:131:in parse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:96:inparse_element'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:96:in parse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:169:inblock in eval_include'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:in each' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:ineval_include'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:146:in parse_include' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:105:inparse_element'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:44:in parse!' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:33:inparse'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config.rb:71:in parse' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config.rb:52:inbuild'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:679:in initialize' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:innew'
from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in <top (required)>' from /usr/share/gems/gems/fluentd-1.15.3/bin/fluentd:15:inrequire'
from /usr/share/gems/gems/fluentd-1.15.3/bin/fluentd:15:in <top (required)>' from /usr/bin/fluentd:23:inload'
from /usr/bin/fluentd:23:in <top (required)>' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:58:inload'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:58:in kernel_load' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:23:inrun'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:492:in exec' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/command.rb:27:inrun'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor.rb:392:indispatch'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:34:in dispatch' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/base.rb:485:instart'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:28:in start' from /usr/local/share/gems/gems/bundler-2.4.9/exe/bundle:45:inblock in <top (required)>'
from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/friendly_errors.rb:117:in with_friendly_errors' from /usr/local/share/gems/gems/bundler-2.4.9/exe/bundle:33:in<top (required)>'
from /usr/local/bin/bundle:23:in load' from /usr/local/bin/bundle:23:in'
What you expected to happen:
The same value file is working in another environment.
How to reproduce it (as minimally and precisely as possible):
not really sure because the reverted config should have work.
Anything else we need to know?:
The change was made in this section under logs:
name_of_pod:
from:
pod: pod_name
container: container_name
timestampExtraction:
format: "%Y-%m-%d %H:%M:%S.%N"
multiline:
firstline: '/^.\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}.\d{3}.$/' --> single quote was removed
separator: "\n"
sourcetype: something
Environment:
Kubernetes version (use kubectl version): 1.25.3
Ruby version (use ruby --version): ruby 2.5.9p229
OS (e.g: cat /etc/os-release): "SUSE Linux Enterprise Server 15 SP4"
Splunk version: Splunk Enterprise 8.2.2.1
Splunk Connect for Kubernetes helm chart version: 1.5.3
What happened: Updated some regex to match the timestamp and upgraded the SCK. However, when the helm was upgraded with a new value file few pods went to Crashloopbackoff.
I reverted the changes and upgraded the helm chart but the problem persisted. I tried to uninstall and install the helm chart again but the issue remains there.
Prior to this change, the logs were being sent from some of the pods. However, multiline was not working so tried to fix the regex.
The following error logs can be seen in pod logs:
bundler: failed to load command: fluentd (/usr/bin/fluentd) /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:92:in'
parse_error!': unexpected end of file in a single quoted string at output.conf line 267,138 (Fluent::ConfigParseError) 266: @type jq_transformer 267: jq '.record | . + (.source | capture("/var/log/pods/(?<pod_uid>[^/]+)/(?<container_name>[^/]+)/(?<container_retry>[0-9]+).log")) ------------------------------------------------------------------------------------------------------------------------------------------^ 268: </filter> from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:131:inscan_single_quoted_string' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:84:inscan_string' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/literal_parser.rb:75:inparse_literal' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:131:inparse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:96:inparse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:96:inparse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:169:inblock in eval_include' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:ineach' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:ineval_include' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:146:inparse_include' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:105:inparse_element' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:44:inparse!' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:33:inparse' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config.rb:71:inparse' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/config.rb:52:inbuild' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:679:ininitialize' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:innew' from /usr/share/gems/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in<top (required)>' from /usr/share/gems/gems/fluentd-1.15.3/bin/fluentd:15:inrequire' from /usr/share/gems/gems/fluentd-1.15.3/bin/fluentd:15:in<top (required)>' from /usr/bin/fluentd:23:inload' from /usr/bin/fluentd:23:in<top (required)>' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:58:inload' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:58:inkernel_load' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli/exec.rb:23:inrun' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:492:inexec' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/command.rb:27:inrun' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:ininvoke_command' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor.rb:392:indispatch' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:34:indispatch' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/vendor/thor/lib/thor/base.rb:485:instart' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/cli.rb:28:instart' from /usr/local/share/gems/gems/bundler-2.4.9/exe/bundle:45:inblock in <top (required)>' from /usr/local/share/gems/gems/bundler-2.4.9/lib/bundler/friendly_errors.rb:117:inwith_friendly_errors' from /usr/local/share/gems/gems/bundler-2.4.9/exe/bundle:33:in<top (required)>' from /usr/local/bin/bundle:23:inload' from /usr/local/bin/bundle:23:in=========================================================
What you expected to happen: The same value file is working in another environment.
How to reproduce it (as minimally and precisely as possible): not really sure because the reverted config should have work.
Anything else we need to know?: The change was made in this section under logs: name_of_pod: from: pod: pod_name container: container_name timestampExtraction: format: "%Y-%m-%d %H:%M:%S.%N"
multiline: firstline: '/^.\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}.\d{3}.$/' --> single quote was removed separator: "\n" sourcetype: something
Environment:
kubectl version): 1.25.3ruby --version): ruby 2.5.9p229cat /etc/os-release): "SUSE Linux Enterprise Server 15 SP4"