Can't Get Splunk Connect for Kubernetes To Send Data To HEC

[BobWieberdink]

I am very new to Splunk and Splunk Connect. We are trying to implement Splunk Connect for Kubernetes. We have gotten the pods running, and it is gathering data to send to the HEC host:

2019-07-22 19:38:41 +0000 [debug]: #0 Received 34 pods 2019-07-22 19:38:41 +0000 [debug]: #0 Going to pull config_maps 2019-07-22 19:38:41 +0000 [debug]: #0 Received 32 config_maps 2019-07-22 19:38:44 +0000 [debug]: #0 taking back chunk for errors. chunk="58e4a3589d3ff809c65d7099990a8d3a" 2019-07-22 19:38:44 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-07-22 19:38:45 +0000 chunk="58e4a3589d3ff809c65d7099990a8d3a" error_class=Net::OpenTimeout error="execution expired" 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in initialize' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:inopen' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in block in connect' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/timeout.rb:103:intimeout' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:935:in connect' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:920:indo_start' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:915:in start' 2019-07-22 19:38:44 +0000 [warn]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:instart'

but is seems when it tries to send to Splunk Cloud, we get the following:

2019-07-22 18:24:03 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2019-07-22 18:24:05 +0000 chunk="58e492321a0231c40c1532cc29712bac" error_class=Net::OpenTimeout error="execution expired"

Here is our current (redacted) values.yaml file we are using to deploy with Helm:

global: logLevel: debug splunk: hec: protocol: https insecureSSL: true host: http-inputs-companyname.splunkcloud.com token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX splunk-kubernetes-logging: journalLogPath: /run/log/journal splunk: hec: indexName: index-log splunk-kubernetes-objects: rbac: create: true serviceAccount: create: true name: index-objects kubernetes: insecureSSL: true objects: core: v1:

• name: pods interval: 30s
• name: namespaces interval: 30s
• name: nodes interval: 30s
• name: services interval: 30s
• name: config_maps interval: 30s
• name: persistent_volumes interval: 30s
• name: service_accounts interval: 30s
• name: persistent_volume_claims interval: 30s
• name: resource_quotas interval: 30s
• name: component_statuses interval: 30s
• name: events mode: watch apps: v1:
• name: deployments interval: 30s
• name: daemon_sets interval: 30s
• name: replica_sets interval: 30s
• name: stateful_sets interval: 30s splunk: hec: indexName: index-objects splunk-kubernetes-metrics: rbac: create: true serviceAccount: create: true name: splunk-kubernetes-metrics splunk: hec: indexName: index-metrics

We have successfully sent Curl requests from the node, pod and container to Splunk Cloud with the following command (redacted): curl -k https://http-inputs-companyname.splunkcloud.com/services/collector/event -H "Authorization: Splunk XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" -d '{"event":"testing with index - inside_container2"}'

and can view the test messages in Splunk.

We believe it to be some sort of configuration issue, but we are at a loss regarding what to change in the values.yaml file we are using to deploy.

Any assistance would be greatly appreciated. Thank you, Bob Wieberdink

[chaitanyaphalak]

Set insecureSSL as false for HEC in your config, also you will have use the port 443.

[BobWieberdink]

YES! - That did it! Thank you, thank you!

Bob Wieberdink