search

splunk / splunk-connect-for-kubernetes

Helm charts associated with kubernetes plug-ins
Apache License 2.0
344 stars 270 forks source link

Pods are going into CrashloopBackoff #848

Closed kavita1205 closed 1 year ago

kavita1205 commented 1 year ago

Hi Team,

I am using latest version of splunk-connect-for-kubernetes helm chart but few pods are going into crashloopbackoff. Can someone please help here.

Error

NAME                                     READY   STATUS             RESTARTS   AGE   IP                NODE                NOMINATED NODE   READINESS GATES
lv-splunk-logging-5q9pf                  0/1     CrashLoopBackOff   12         47m   192.168.54.241    las2-mlgpu31        <none>           <none>
lv-splunk-logging-nzzld                  0/1     CrashLoopBackOff   13         47m   192.168.97.152    las2-mlgpu18        <none>           <none>
lv-splunk-logging-qjfgw                  0/1     CrashLoopBackOff   13         47m   192.168.178.41    aws-mlgpu03         <none>           <none>
lv-splunk-logging-zmvxp                  0/1     CrashLoopBackOff   13         47m   192.168.11.174    las2-mlgpu11        <none>           <none>

kubectl logs :

2023-02-06 14:42:08 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-02-06 14:42:08 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-02-06 14:42:08 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-jq' version '0.5.1'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.0'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-02-06 14:42:08 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2023-02-06 14:42:08 +0000 [INFO]: Reading bearer token from /var/run/secrets/kubernetes.io/serviceaccount/token
2023-02-06 14:42:11 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Invalid Kubernetes API v1 endpoint https://10.96.0.1:443/api: Timed out connecting to server"

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

hvaghani221 commented 1 year ago

Hi @kavita1205, can you share your values.yaml file? Also, did you make any changes to the template files?

kavita1205 commented 1 year ago

Hi @harshit-splunk , I have fixed this issue by fixing calico networking pods. Now, I am facing issue i.e. logs are not being generated in SCK. For the same, I have raised issue here in #849.