search

signalfx / splunk-otel-collector-chart

Splunk OpenTelemetry Collector for Kubernetes
Apache License 2.0
122 stars 151 forks source link

Exporting failed. Will retry the request after interval #397

Closed beth2016 closed 2 years ago

beth2016 commented 2 years ago

In the splunk-otel-collector-agent logs for WINDOWS, I am getting these errors every few seconds. What am I missing? thanks.

go.opentelemetry.io/collector/exporter/exporterhelper.onTemporaryFailure /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/queued_retry_inmemory.go:93 go.opentelemetry.io/collector/exporter/exporterhelper.(retrySender).send /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/queued_retry.go:199 go.opentelemetry.io/collector/exporter/exporterhelper.(logsExporterWithObservability).send /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/logs.go:133 go.opentelemetry.io/collector/exporter/exporterhelper.(queuedRetrySender).start.func1 /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/queued_retry_inmemory.go:105 go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/internal/bounded_memory_queue.go:99 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(boundedMemoryQueue).StartConsumers.func2 /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.43.1/exporter/exporterhelper/internal/bounded_memory_queue.go:78 2022-03-04T20:48:48.666Z error exporterhelper/queued_retry_inmemory.go:93 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "name": "splunk_hec/platform_logs", "error": "max elapsed time expired HTTP 403 \"Forbidden\"", "dropped_items": 4}

matthewmodestino commented 2 years ago

Looks like a 403 forbidden on the HEC URL. I would double check this section:

https://github.com/signalfx/splunk-otel-collector-chart/blob/75cfd0b637fee138191bf48b7cb323d1440b70c4/helm-charts/splunk-otel-collector/values.yaml#L25-L32

403 usually means wrong token

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/TroubleshootHTTPEventCollector#:~:text=4,Invalid%20token

beth2016 commented 2 years ago

@matthewmodestino That is it. Thanks so much.