panic: runtime error: invalid memory address or nil pointer dereference

[qfframb]

Describe the bug fluent-bit pod CrashLoopBackOff

To Reproduce

Rubular link if applicable: Example log message if applicable: level=info caller=out_loki.go:113 ts=2021-12-21T09:18:51.293142224Z id=1 [flb-go]="Starting fluent-bit-go-loki" version="(version=, branch=, revision=)" level=info caller=out_loki.go:115 ts=2021-12-21T09:18:51.294007807Z id=1 [flb-go]="provided parameter" URL=http://loki.garden.svc:3100/loki/api/v1/push level=info caller=out_loki.go:116 ts=2021-12-21T09:18:51.294022973Z id=1 [flb-go]="provided parameter" TenantID= level=info caller=out_loki.go:117 ts=2021-12-21T09:18:51.294042324Z id=1 [flb-go]="provided parameter" BatchWait=1m0s E1221 09:18:51.293278 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 58 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x7fa5344b2020, 0x7fa535320590) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89 panic(0x7fa5344b2020, 0x7fa535320590) /usr/local/go/src/runtime/panic.go:969 +0x1b9 github.com/gardener/logging/pkg/controller.getShootState(0x0, 0x7fa534742b40, 0xc000085560) /go/src/github.com/gardener/logging/pkg/controller/utils.go:74 +0x2c9 github.com/gardener/logging/pkg/controller.(controller).updateControllerClientState(0xc000776380, 0x7fa53475f780, 0xc00047cee0, 0x0) /go/src/github.com/gardener/logging/pkg/controller/client.go:118 +0x2d github.com/gardener/logging/pkg/controller.(controller).createControllerClient(0xc000776380, 0xc0002f01e0, 0x1c, 0x0) /go/src/github.com/gardener/logging/pkg/controller/client.go:85 +0x3ef github.com/gardener/logging/pkg/controller.(controller).addFunc(0xc000776380, 0x7fa5346a59e0, 0xc0003ee320) /go/src/github.com/gardener/logging/pkg/controller/controller.go:125 +0x10c k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...) /go/src/github.com/gardener/logging/vendor/k8s.io/client-go/tools/cache/controller.go:198 k8s.io/client-go/tools/cache.(processorListener).run.func1.1(0x0, 0x0, 0x0) /go/src/github.com/gardener/logging/vendor/k8s.io/client-go/tools/cache/shared_informer.go:658 +0xc2 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc00076be18, 0x0, 0x0) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:292 +0x57 k8s.io/client-go/tools/cache.(processorListener).run.func1() /go/src/github.com/gardener/logging/vendor/k8s.io/client-go/tools/cache/shared_informer.go:652 +0x7b k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000337760) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x62 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00076bf60, 0xdf8475800, 0x0, 0x1, 0xc0001cc540) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 k8s.io/client-go/tools/cache.(processorListener).run(0xc000776400) /go/src/github.com/gardener/logging/vendor/k8s.io/client-go/tools/cache/shared_informer.go:650 +0x9d k8s.io/apimachinery/pkg/util/wait.(Group).Start.func1(0xc000554290, 0xc000639080) /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x53 created by k8s.io/apimachinery/pkg/util/wait.(Group).Start /go/src/github.com/gardener/logging/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x65 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x2c8 pc=0x7fa534076209]

version: fluent-bit: 1.8.7 fluent-bit-to-loki: v0.38.0 loki: 2.2.1

config: filter-kubernetes.conf: |-

Lua filter to add the tag into the record

[FILTER] Name lua Match kubernetes.* script add_tag_to_record.lua call add_tag_to_record

Systemd Filters

[FILTER] Name record_modifier Match journald.docker Record hostname ${NODE_NAME} Record unit docker

[FILTER] Name record_modifier Match journald.containerd Record hostname ${NODE_NAME} Record unit containerd

[FILTER] Name record_modifier Match journald.kubelet Record hostname ${NODE_NAME} Record unit kubelet

[FILTER] Name record_modifier Match journald.cloud-config-downloader* Record hostname ${NODE_NAME} Record unit cloud-config-downloader

[FILTER] Name record_modifier Match journald.docker-monitor Record hostname ${NODE_NAME} Record unit docker-monitor

[FILTER] Name record_modifier Match journald.containerd-monitor Record hostname ${NODE_NAME} Record unit containerd-monitor

[FILTER] Name record_modifier Match journald.kubelet-monitor Record hostname ${NODE_NAME} Record unit kubelet-monitor

Shoot controlplane filters

[FILTER] Name parser Match kubernetes.kube-state-metricskube-state-metrics* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.addons-kubernetes-dashboardkubernetes-dashboard* Key_Name log Parser kubernetesdashboardParser Reserve_Data True

System components filters

[FILTER] Name parser Match kubernetes.kube-proxykube-proxy* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.addons-nginx-ingress-controllernginx-ingress-controller* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.vpn-shootvpn-shoot* Key_Name log Parser vpnshootParser Reserve_Data True

[FILTER] Name parser Match kubernetes.node-exporternode-exporter* Key_Name log Parser nodeexporterParser Reserve_Data True

[FILTER] Name parser Match kubernetes.node-problem-detectornode-problem-detector* Key_Name log Parser kubeapiserverParser Reserve_Data True

Garden filters

[FILTER] Name parser Match kubernetes.alertmanageralertmanager* Key_Name log Parser alertmanagerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.prometheusprometheus* Key_Name log Parser alertmanagerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.grafanagrafana* Key_Name log Parser grafanaParser Reserve_Data True

[FILTER] Name parser Match kubernetes.vpa- Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.hvpa-controllerhvpa-controller* Key_Name log Parser kubeapiserverParser Reserve_Data True

Extension filters

[FILTER] Name parser Match kubernetes.gardener-extension Key_Name log Parser extensionsParser Reserve_Data True

[FILTER] Name modify Match kubernetes.gardener-extension Rename level severity Rename msg log Rename logger source

Extensions

[FILTER] Name parser Match kubernetes.dependency-watchdogdependency-watchdog* Key_Name log Parser dependencyWatchdogParser Reserve_Data True

[FILTER] Name parser Match kubernetes.gardener-seed-admission-controllergardener-seed-admission-controller* Key_Name log Parser gsacParser Reserve_Data True

[FILTER] Name parser Match kubernetes.gardener-resource-managergardener-resource-manager* Key_Name log Parser gardenerResourceManagerParser Reserve_Data True

[FILTER] Name modify Match kubernetes.gardener-resource-managergardener-resource-manager* Rename level severity Rename msg log Rename logger source

[FILTER] Name parser Match kubernetes.etcdetcd* Key_Name log Parser etcdParser Reserve_Data True

[FILTER] Name parser Match kubernetes.etcdbackup-restore* Key_Name log Parser backupRestoreParser Reserve_Data True

[FILTER] Name parser Match kubernetes.cluster-autoscalercluster-autoscaler* Key_Name log Parser clusterAutoscalerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.kube-apiserverkube-apiserver* Key_Name log Parser kubeAPIServerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.kube-apiservervpn-seed* Key_Name log Parser vpnSeedParser Reserve_Data True

[FILTER] Name parser Match kubernetes.kube-schedulerkube-scheduler* Key_Name log Parser kubeSchedulerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.kube-controller-managerkube-controller-manager* Key_Name log Parser kubeControllerManagerParser Reserve_Data True

[FILTER] Name parser Match kubernetes.corednscoredns* Key_Name log Parser coreDNSParser1 Parser coreDNSParser2 Reserve_Data True

[FILTER] Name parser Match kubernetes.metrics-servermetrics-server* Key_Name log Parser metricsServerParser Reserve_Data True

[FILTER] Name modify Match kubernetes.* Condition Key_valuematches tag ^kubernetes.var.log.containers.(kube-apiserver|vpa-exporter|vpa-recommender|vpa-admission-controller|cluster-autoscaler|kube-scheduler|kube-controller-manager)-.+? Add gardener_multitenant_id operator;user

[FILTER] Name parser Match kubernetes.calico-nodecalico-node* Key_Name log Parser caliconodeParser Reserve_Data True

[FILTER] Name parser Match kubernetes.cloud-controller-manageralicloud-cloud-controller-manager* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.machine-controller-manageralicloud-machine-controller-manager* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.machine-controller-managermachine-controller-manager-provider-alicloud* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name parser Match kubernetes.csi-plugin-controlleralicloud-csi* Key_Name log Parser kubeapiserverParser Reserve_Data True

[FILTER] Name modify Match kubernetes.* Condition Key_valuematches tag ^kubernetes.var.log.containers.(cloud-controller-manager-.+?.+?alicloud-cloud-controller-manager|csi-plugin-controller-.+?.+?_alicloud-csi) Add gardener_multitenant_id operator;user

[FILTER] Name modify Match kubernetes.* Condition Key_valuematches tag ^kubernetes.var.log.containers.machine-controller-manager-.+?.+?_(alicloud-machine-controller-manager|machine-controller-manager-provider-alicloud) Add gardener_multitenant_id operator;user

Scripts

[FILTER] Name lua Match kubernetes.* script modify_severity.lua call cb_modify

fluent-bit.conf: |-

Service section

[SERVICE] Flush 30 Daemon Off Log_Level info Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_PORT 2020

@INCLUDE input.conf @INCLUDE filter-kubernetes.conf @INCLUDE output.conf input.conf: |-

Input section

[INPUT] Name tail Tag kubernetes. Path /var/log/containers/.log Exclude_Path _garden_fluent-bit-.log,_garden_loki-.log Parser docker DB /var/log/flb_kube.db DB.sync full read_from_head true Skip_Long_Lines On Mem_Buf_Limit 30MB Refresh_Interval 10 Ignore_Older 1800s

[INPUT] Name systemd Tag journald.docker Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=docker.service

[INPUT] Name systemd Tag journald.kubelet Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=kubelet.service

[INPUT] Name systemd Tag journald.containerd Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=containerd.service

[INPUT] Name systemd Tag journald.cloud-config-downloader Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=cloud-config-downloader.service

[INPUT] Name systemd Tag journald.docker-monitor Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=docker-monitor.service

[INPUT] Name systemd Tag journald.containerd-monitor Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=containerd-monitor.service

[INPUT] Name systemd Tag journald.kubelet-monitor Path %%JOURNALD_PATH%% Read_From_Tail True Systemd_Filter _SYSTEMD_UNIT=kubelet-monitor.service kubernetes_label_map.json: |- { "kubernetes": {"container_name":"container_name","docker_id":"docker_id","namespace_name":"namespace_name","pod_name":"pod_name"} , "severity": "severity" } modify_severity.lua: |- function cb_modify(tag, timestamp, record) local unified_severity = cb_modify_unify_severity(record)

if not unified_severity then return 0, 0, 0 end

return 1, timestamp, record end

function cb_modify_unify_severity(record) local modified = false local severity = record["severity"] if severity == nil or severity == "" then return modified end

severity = trim(severity):upper()

if severity == "I" or severity == "INF" or severity == "INFO" then record["severity"] = "INFO" modified = true elseif severity == "W" or severity == "WRN" or severity == "WARN" or severity == "WARNING" then record["severity"] = "WARN" modified = true elseif severity == "E" or severity == "ERR" or severity == "ERROR" or severity == "EROR" then record["severity"] = "ERR" modified = true elseif severity == "D" or severity == "DBG" or severity == "DEBUG" then record["severity"] = "DBG" modified = true elseif severity == "N" or severity == "NOTICE" then record["severity"] = "NOTICE" modified = true elseif severity == "F" or severity == "FATAL" then record["severity"] = "FATAL" modified = true end

return modified end

function trim(s) return (s:gsub("^%s(.-)%s$", "%1")) end

output.conf: |-

Output section

[Output] Name gardenerloki Match kubernetes.* Url http://loki.garden.svc:3100/loki/api/v1/push LogLevel info BatchWait 40s BatchSize 30720 Labels {origin="seed"} LineFormat json SortByTimestamp true DropSingleKey false AutoKubernetesLabels false LabelSelector gardener.cloud/role:shoot RemoveKeys kubernetes,stream,time,tag,gardenuser LabelMapPath /fluent-bit/etc/kubernetes_label_map.json DynamicHostPath {"kubernetes": {"namespace_name": "namespace"}} DynamicHostPrefix http://loki. DynamicHostSuffix .svc:3100/loki/api/v1/push DynamicHostRegex ^shoot- DynamicTenant user gardenuser user MaxRetries 3 Timeout 10s MinBackoff 30s Buffer true BufferType dque QueueDir /fluent-bit/buffers/seed QueueSegmentSize 300 QueueSync normal QueueName gardener-kubernetes-operator FallbackToTagWhenMetadataIsMissing true TagKey tag DropLogEntryWithoutK8sMetadata true SendDeletedClustersLogsToDefaultClient true CleanExpiredClientsPeriod 1h ControllerSyncTimeout 120s NumberOfBatchIDs 5 TenantID operator

[Output] Name gardenerloki Match journald.* Url http://loki.garden.svc:3100/loki/api/v1/push LogLevel info BatchWait 60s BatchSize 30720 Labels {origin="seed-journald"} LineFormat json SortByTimestamp true DropSingleKey false RemoveKeys kubernetes,stream,hostname,unit LabelMapPath /fluent-bit/etc/systemd_label_map.json MaxRetries 3 Timeout 10s MinBackoff 30s Buffer true BufferType dque QueueDir /fluent-bit/buffers QueueSegmentSize 300 QueueSync normal QueueName seed-journald NumberOfBatchIDs 5 parsers.conf: |-

Custom parsers

[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On

Command | Decoder | Field | Optional Action

=============|==================|=================

Decode_Field_As escaped log

[PARSER] Name kubeapiserverParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name alertmanagerParser Format regex Regex ^level=(?\w+)\s+ts=(?\d{4}-\d{2}-\d{2}[Tt].[zZ])\s+caller=(?[^\s]+)\s+(?.*) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER] Name vpnshootParser Format regex Regex ^(?[^0-9]\d{1,2}\s+[^\s]+\s+\d{4})\s+(?.) Time_Key time Time_Format %a %b%t%d %H:%M:%S %Y

[PARSER] Name kubernetesdashboardParser Format regex Regex ^(?\d{4}\/\d{2}\/\d{2}\s+[^\s])\s+(?.) Time_Key time Time_Format %Y/%m/%d %H:%M:%S

[PARSER] Name nodeexporterParser Format regex Regex ^time="(?\d{4}-\d{2}-\d{2}T[^"])"\s+level=(?\w+)\smsg="(?.)"\s+source="(?.*)" Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER] Name grafanaParser Format regex Regex ^t=(?\d{4}-\d{2}-\d{2}T[^ ])\s+lvl=(?\w+)\smsg="(?.)"\s+logger=(?.*) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S%z

[PARSER] Name extensionsParser Format json Time_Key ts Time_Format %Y-%m-%dT%H:%M:%S

[PARSER] Name dependencyWatchdogParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name gsacParser Format regex Regex ^time="(?\d{4}-\d{2}-\d{2}T[^"])"\s+level=(?\w+)\smsg="(?.)" Time_Key time Time_Format %Y-%m-%dT%H:%M:%S%z

[PARSER] Name gardenerResourceManagerParser Format json Time_Key ts Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER] Name etcdParser Format regex Regex ^(?\d{4}-\d{2}-\d{2}\s+[^ ])\s+(?\w+)\s+|\s+(?[^ :]):\s+(?.*) Time_Key time Time_Format %Y-%m-%d %H:%M:%S.%L

[PARSER] Name backupRestoreParser Format regex Regex ^time="(?\d{4}-\d{2}-\d{2}T[^"])"\s+level=(?\w+)\smsg="(?.)" Time_Key time Time_Format %Y-%m-%dT%H:%M:%S%z

[PARSER] Name clusterAutoscalerParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name kubeAPIServerParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name vpnSeedParser Format regex Regex ^(?[^0-9]\d{1,2}\s+[^\s]+\s+\d{4})\s+(?.) Time_Key time Time_Format %a %b%t%d %H:%M:%S %Y

[PARSER] Name kubeSchedulerParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name kubeControllerManagerParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name coreDNSParser1 Format regex Regex ^(?\d{4}-\d{2}-\d{2}[Tt].[zZ])\s+[(?\w[^]])]\s+(?.*) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER] Name coreDNSParser2 Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name metricsServerParser Format regex Regex ^(?\w)(?\d{4} [^\s])\s+(?\d+)\s+(?[^ ]]+)] (?.)$ Time_Key time Time_Format %m%d %H:%M:%S.%L

[PARSER] Name caliconodeParser Format regex Regex ^(?\d{4}-\d{2}-\d{2}\s+[^ ])\s+[(?\w)][(?\d+)]\s+(?[^:]):\s+(?.) Time_Key time Time_Format %Y-%m-%d %H:%M:%S.%L

plugin.conf: |- [PLUGINS] Path /fluent-bit/plugins/out_loki.so

[qfframb]

issue solved by ourself