search

signalfx / splunk-otel-collector-chart

Splunk OpenTelemetry Collector for Kubernetes
Apache License 2.0
116 stars 146 forks source link

Need help on connection error #987

Closed xxPhuNguyenxx closed 11 months ago

xxPhuNguyenxx commented 11 months ago

I'm currently trying to setup our onprem Openshift to use Splunk OpenTelemetry but it's throwing connection errors as seen below

here's our values.yaml file

clusterName: ocp4-onprem-sbx
splunkPlatform:
  token: xxxxxxxxxxxxx
  endpoint: https://SPLUNK_HF/services/collector/event
  index: sotl
  metricsIndex: sotm
  logsEnabled: true
  metricsEnabled: false
  tracesEnabled: false
  maxConnections: 10 # Maximum HTTP connections to use simultaneously when sending data.
  disableCompression: false # Whether to disable gzip compression over HTTP. Defaults to true.
  timeout: 10s # HTTP timeout when sending data. Defaults to 10s.
  idleConnTimeout: 5s # Idle connection timeout. defaults to 10s
  insecureSkipVerify: false `
  retryOnFailure:
    enabled: true
    initialInterval: 30s # Time to wait after the first failure before retrying; ignored if enabled is false. Defaults to 5s 
    maxInterval: 60s # The upper bound on backoff; ignored if enabled is false. Default is 30s
    maxElapsedTime: 600s # The maximum amount of time spent trying to send a batch; ignored if enabled is false. Default is 300s
logsEngine: otel
logsCollection:
  containers:
    enabled: true
    excludeAgentLogs: true
cloudProvider: ""
distribution: "openshift"
environment: sandbox

Log: 

[AA11111@vaple001 ~]$ oc project splunk-otel
Now using project "splunk-otel" on server "https://api.ocp.cluster.das:6443".
[AA11111@vaple001 ~]$ oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
ocp4-onprem-sbx-splunk-otel-collector-agent-2992k   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-lsdnh   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-p5gq6   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-pvxnz   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-qkmxz   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-scwjg   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-t8fk9   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-t9pvx   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-w562x   1/1     Running   0          4d22h
ocp4-onprem-sbx-splunk-otel-collector-agent-xh5jb   1/1     Running   0          4d22h
[AA11111@vaple001 ~]$ oc logs ocp4-onprem-sbx-splunk-otel-collector-agent-2992k
2023/10/18 19:16:01 settings.go:436: Set config to [/conf/relay.yaml]
2023/10/18 19:16:01 settings.go:489: Set ballast to 165 MiB
2023/10/18 19:16:01 settings.go:505: Set memory limit to 450 MiB
2023/10/18 19:16:01 settings.go:396: set "SPLUNK_LISTEN_INTERFACE" to "0.0.0.0"
2023-10-18T19:16:01.185Z        info    service@v0.86.0/telemetry.go:84 Setting up own telemetry...
2023-10-18T19:16:01.186Z        info    service@v0.86.0/telemetry.go:201        Serving Prometheus metrics     {"address": "0.0.0.0:8889", "level": "Basic"}
2023-10-18T19:16:01.268Z        info    kube/client.go:107      k8s filtering   {"kind": "processor", "name": "k8sattributes", "pipeline": "logs", "labelSelector": "", "fieldSelector": "spec.nodeName=vaple001"}
2023-10-18T19:16:01.268Z        info    memorylimiterprocessor@v0.86.0/memorylimiter.go:102     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 450, "spike_limit_mib": 90, "check_interval": 2}
2023-10-18T19:16:01.270Z        info    service@v0.86.0/service.go:138  Starting otelcol...     {"Version": "v0.86.0", "NumCPU": 16}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:31     Starting extensions...
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:34     Extension is starting...        {"kind": "extension", "name": "file_storage"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:38     Extension started.      {"kind": "extension", "name": "file_storage"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:34     Extension is starting...        {"kind": "extension", "name": "health_check"}
2023-10-18T19:16:01.270Z        info    healthcheckextension@v0.86.0/healthcheckextension.go:35 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2023-10-18T19:16:01.270Z        warn    internal@v0.86.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:38     Extension started.      {"kind": "extension", "name": "health_check"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:34     Extension is starting...        {"kind": "extension", "name": "k8s_observer"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:38     Extension started.      {"kind": "extension", "name": "k8s_observer"}
2023-10-18T19:16:01.270Z        info    extensions/extensions.go:34     Extension is starting...        {"kind": "extension", "name": "memory_ballast"}
2023-10-18T19:16:01.276Z        info    ballastextension@v0.86.0/memory_ballast.go:41   Setting memory ballast {"kind": "extension", "name": "memory_ballast", "MiBs": 165}
2023-10-18T19:16:01.368Z        info    extensions/extensions.go:38     Extension started.      {"kind": "extension", "name": "memory_ballast"}
2023-10-18T19:16:01.368Z        info    extensions/extensions.go:34     Extension is starting...        {"kind": "extension", "name": "zpages"}
2023-10-18T19:16:01.368Z        info    zpagesextension@v0.86.0/zpagesextension.go:53   Registered zPages span processor on tracer provider     {"kind": "extension", "name": "zpages"}
2023-10-18T19:16:01.368Z        info    zpagesextension@v0.86.0/zpagesextension.go:63   Registered Host's zPages{"kind": "extension", "name": "zpages"}
2023-10-18T19:16:01.369Z        info    zpagesextension@v0.86.0/zpagesextension.go:75   Starting zPages extension       {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2023-10-18T19:16:01.369Z        info    extensions/extensions.go:38     Extension started.      {"kind": "extension", "name": "zpages"}
2023-10-18T19:16:01.369Z        info    internal/resourcedetection.go:125       began detecting resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2023-10-18T19:16:01.399Z        info    internal/resourcedetection.go:139       detected resource information  {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "resource": {"host.name":"va33slvocp007","os.type":"linux"}}
2023-10-18T19:16:01.467Z        info    adapter/receiver.go:45  Starting stanza receiver        {"kind": "receiver", "name": "filelog", "data_type": "logs"}
2023-10-18T19:16:01.467Z        info    fileconsumer/file.go:354        Resuming from previously known offset(s). 'start_at' setting is not applicable. {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-10-18T19:16:01.472Z        warn    internal@v0.86.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "receiver", "name": "otlp", "data_type": "logs", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-18T19:16:01.472Z        info    otlpreceiver@v0.86.0/otlp.go:83 Starting GRPC server    {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4317"}
2023-10-18T19:16:01.473Z        warn    internal@v0.86.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "receiver", "name": "otlp", "data_type": "logs", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-18T19:16:01.473Z        info    otlpreceiver@v0.86.0/otlp.go:101        Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4318"}
2023-10-18T19:16:01.473Z        info    healthcheck/handler.go:132      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
2023-10-18T19:16:01.473Z        info    service@v0.86.0/service.go:161  Everything is ready. Begin running and processing data.
2023-10-18T19:16:01.679Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55566.log"}
2023-10-18T19:17:16.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19976.log"}
2023-10-18T19:20:43.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55567.log"}
2023-10-18T19:22:20.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19977.log"}
2023-10-18T19:25:56.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55568.log"}
2023-10-18T19:27:25.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19978.log"}
2023-10-18T19:31:22.078Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55569.log"}
2023-10-18T19:32:32.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19979.log"}
2023-10-18T19:36:33.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55570.log"}
2023-10-18T19:37:37.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19980.log"}
2023-10-18T19:41:47.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55571.log"}
2023-10-18T19:42:52.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19981.log"}
2023-10-18T19:47:10.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55572.log"}
2023-10-18T19:47:56.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19982.log"}
2023-10-18T19:52:22.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55573.log"}
2023-10-18T19:52:30.279Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55574.log"}
2023-10-18T19:53:09.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19983.log"}
2023-10-18T19:57:45.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55575.log"}
2023-10-18T19:58:20.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19984.log"}
2023-10-18T20:03:01.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55576.log"}
2023-10-18T20:03:35.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19985.log"}
2023-10-18T20:08:24.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55577.log"}
2023-10-18T20:08:41.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19986.log"}
2023-10-18T20:13:36.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55578.log"}
2023-10-18T20:13:48.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19987.log"}
2023-10-18T20:18:50.878Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55579.log"}
2023-10-18T20:19:04.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19988.log"}
2023-10-18T20:19:43.952Z        info    exporterhelper/retry_sender.go:177      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post \"https://SPLUNK_HF/services/collector/event\": read tcp 30.118.45.9:59688->30.254.254.155:443: read: connection reset by peer", "interval": "17.987634619s"}
2023-10-18T20:24:12.881Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55580.log"}
2023-10-18T20:29:19.880Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19990.log"}
2023-10-18T20:29:24.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55581.log"}
2023-10-18T20:30:00.323Z        info    exporterhelper/retry_sender.go:177      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post \"https://SPLUNK_HF/services/collector/event\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)", "interval": "37.248737291s"}
2023-10-18T20:34:30.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19991.log"}
2023-10-18T20:34:41.878Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55582.log"}
2023-10-18T20:39:40.080Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19992.log"}
2023-10-18T20:39:52.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55583.log"}
2023-10-18T20:44:48.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19993.log"}
2023-10-18T20:45:06.078Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55584.log"}
2023-10-18T20:50:05.080Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19994.log"}
2023-10-18T20:50:23.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55585.log"}
2023-10-18T20:55:14.880Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19995.log"}
2023-10-18T20:55:41.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55586.log"}
2023-10-18T20:55:45.278Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55587.log"}
2023-10-18T21:00:24.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19996.log"}
2023-10-18T21:01:07.079Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55588.log"}
2023-10-18T21:03:11.463Z        info    exporterhelper/retry_sender.go:177      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post \"https://SPLUNK_HF/services/collector/event\": read tcp 30.118.45.9:33980->30.254.254.155:443: read: connection reset by peer", "interval": "29.476499952s"}
2023-10-18T21:05:28.080Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19997.log"}
2023-10-18T21:06:17.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/anusha-test_custom-athem-registry-quay-app-6b4447ff86-256c8_c788a526-ddd0-4f57-beb2-acd5778d5350/quay-app/55589.log"}
2023-10-18T21:07:16.866Z        info    exporterhelper/retry_sender.go:177      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post \"https://SPLUNK_HF/services/collector/event\": EOF", "interval": "32.002757907s"}
2023-10-18T21:07:30.299Z        info    exporterhelper/retry_sender.go:177      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post \"https://SPLUNK_HF/services/collector/event\": read tcp 30.118.45.9:39220->30.254.254.155:443: read: connection reset by peer", "interval": "22.898325411s"}
2023-10-18T21:10:42.879Z        info    fileconsumer/file.go:194        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/geo-quay_geo-anthem-registry-quay-app-c7fd4dc45-7p4bn_672d01ca-061e-4070-8daa-c0b84621887e/quay-app/19998.log"}
atoulme commented 11 months ago

Edited the token out. Please rotate it if possible.

Please open a support case as this is in all likelihood a problem involving your heavy forwarder as well, and support will want to look at your heavy forwarder logs, investigate your load balancer, etc.

xxPhuNguyenxx commented 11 months ago

Thanks for the token catch. ill submit a ticket to splunk support