signalfx / splunk-otel-collector-chart

Splunk OpenTelemetry Collector for Kubernetes
Apache License 2.0
119 stars 148 forks source link

[0.54.0/ 0.57.2] splunk-otel-agent raise error: Post http://splunk-otel:9943/v2/datapoint context deadline exceeded #538

Closed johnzheng1975 closed 2 years ago

johnzheng1975 commented 2 years ago

splunk-otel-agent raise error: Post http://splunk-otel:9943/v2/datapoint context deadline exceeded (Client.Timeout exceeded while awaiting headers)


2022-09-22T11:04:56.275Z    error   exporterhelper/queued_retry_inmemory.go:107 Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "dropped_items": 310}
go.opentelemetry.io/collector/exporter/exporterhelper.onTemporaryFailure
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry_inmemory.go:107
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:04:56.767Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "24.973021824s"}
2022-09-22T11:04:59.553Z    error   exporterhelper/queued_retry_inmemory.go:107 Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "max elapsed time expired Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 53}
go.opentelemetry.io/collector/exporter/exporterhelper.onTemporaryFailure
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry_inmemory.go:107
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.54.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:05:04.418Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "10.348376537s"}
2022-09-22T11:05:08.772Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "32.152169892s"}
2022-09-22T11:05:10.657Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "34.746028558s"}
2022-09-22T11:05:11.096Z    info    fileconsumer/file.go:175    Started watching file   {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/datamanagement-dev_devicevalve-dffdcb8f6-msnnb_04e0a98b-e694-4139-b3a2-b9a624492afd/service/426.log"}
2022-09-22T11:05:14.520Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "7.353741964s"}
2022-09-22T11:05:21.287Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "29.393344337s"}
2022-09-22T11:05:21.742Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "15.733686378s"}
2022-09-22T11:05:26.877Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "15.321513161s"}
2022-09-22T11:05:33.791Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "26.296220406s"}
2022-09-22T11:05:40.163Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "27.871300222s"}
2022-09-22T11:05:40.925Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: i/o timeout\"", "interval": "34.881083827s"}
2022-09-22T11:05:48.389Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "41.34697759s"}
2022-09-22T11:05:52.762Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "39.231795481s"}
2022-09-22T11:05:52.892Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "28.308642004s"}
2022-09-22T11:06:04.077Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "18.686950431s"}
2022-09-22T11:06:05.087Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "29.491697737s"}
2022-09-22T11:06:05.205Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.328706571s"}
2022-09-22T11:06:16.634Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "29.389087363s"}
2022-09-22T11:06:18.706Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "15.278439134s"}
2022-09-22T11:06:19.593Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "44.086908919s"}
2022-09-22T11:06:27.777Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "17.496669534s"}
johnzheng1975 commented 2 years ago

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: splunk-otel
  namespace: infra
spec:
  interval: 10m
  url: https://signalfx.github.io/splunk-otel-collector-chart
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: splunk-otel
  namespace: infra
spec:
  interval: 5m # The interval at which to reconcile the Helm release
  releaseName: splunk-otel
  chart:
    spec:
      chart: splunk-otel-collector
      version: 0.54.0
      sourceRef:
        kind: HelmRepository
        name: splunk-otel
  valuesFrom:
  - kind: Secret
    name: splunk-token
  - kind: Secret
    name: splunk-otel-token
  values:
    nameOverride: splunk-otel
    clusterName: ${ENVIRONMENT}${ENVIRONMENT_SUFFIX}
    environment: ${ENVIRONMENT}
    extraAttributes:
      fromLabels:
      - key: app
        tag_name: label_app
        from: pod
      - key: release
        tag_name: label_release
        from: pod
      - key: type
        tag_name: label_type
        from: pod
      custom:
      - name: cluster_codename
        value: ${CLUSTER_NAME}
      - name: cluster_region
        value: ${AWS_REGION}
      - name: project
        value: onecloud
    splunkPlatform:
      endpoint: https://${SPLUNK_HOST_NAME}:8088/services/collector
      insecureSkipVerify: true
      timeout: 30s
      index: infra-${ENVIRONMENT}
      fieldNameConvention:
        renameFieldsSck: true
        keepOtelConvention: false
    logsEngine: otel
    splunkObservability:
      realm: us1
      #metricsEnabled: false
    cloudProvider: aws
    distribution: eks
    gateway:
      enabled: true
      replicaCount: ${SPLUNK_OTEL_GATEWAY_REPLICAS}
      resources:
        requests:
          cpu: ${SPLUNK_OTEL_GATEWAY_REQUEST_CPU}
    clusterReceiver:
      resources:
        limits:
          cpu: ${SPLUNK_OTEL_RECEIVER_LIMIT_CPU}
          memory: ${SPLUNK_OTEL_RECEIVER_LIMIT_MEMORY}
    agent:
      resources:
        requests:
          memory: ${SPLUNK_OTEL_AGNET_REQUEST_MEMORY}
        limits:
          memory: ${SPLUNK_OTEL_AGENT_LIMIT_MEMORY}

``
johnzheng1975 commented 2 years ago

logs is:

(newenv):  oc-dev-blue@PS0IMMjohn:~/tmp/logmissing$ k logs -n infra                splunk-otel-agent-kvmfs
2022/09/22 11:11:52 main.go:225: Set config to [/conf/relay.yaml]
2022/09/22 11:11:52 main.go:291: Set ballast to 1351 MiB
2022/09/22 11:11:52 main.go:305: Set memory limit to 3686 MiB
2022-09-22T11:11:52.534Z    info    service/telemetry.go:103    Setting up own telemetry...
2022-09-22T11:11:52.535Z    info    service/telemetry.go:138    Serving Prometheus metrics  {"address": "0.0.0.0:8889", "level": "basic"}
2022-09-22T11:11:52.535Z    info    memorylimiterprocessor/memorylimiter.go:113 Memory limiter configured   {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "limit_mib": 3686, "spike_limit_mib": 737, "check_interval": 2}
2022-09-22T11:11:52.538Z    info    signalfxexporter@v0.57.2/factory.go:91  Correlation tracking enabled    {"kind": "exporter", "data_type": "traces", "name": "signalfx", "endpoint": "http://splunk-otel:6060"}
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:42 Starting extensions...
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:45 Extension is starting...    {"kind": "extension", "name": "health_check"}
2022-09-22T11:11:52.637Z    info    healthcheckextension@v0.57.2/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:49 Extension started.  {"kind": "extension", "name": "health_check"}
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:45 Extension is starting...    {"kind": "extension", "name": "k8s_observer"}
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:49 Extension started.  {"kind": "extension", "name": "k8s_observer"}
2022-09-22T11:11:52.637Z    info    extensions/extensions.go:45 Extension is starting...    {"kind": "extension", "name": "memory_ballast"}
2022-09-22T11:11:52.830Z    info    ballastextension/memory_ballast.go:52   Setting memory ballast  {"kind": "extension", "name": "memory_ballast", "MiBs": 1351}
2022-09-22T11:11:52.832Z    info    extensions/extensions.go:49 Extension started.  {"kind": "extension", "name": "memory_ballast"}
2022-09-22T11:11:52.832Z    info    extensions/extensions.go:45 Extension is starting...    {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z    info    zpagesextension/zpagesextension.go:64   Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z    info    zpagesextension/zpagesextension.go:74   Registered Host's zPages    {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z    info    zpagesextension/zpagesextension.go:86   Starting zPages extension   {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2022-09-22T11:11:52.832Z    info    extensions/extensions.go:49 Extension started.  {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z    info    extensions/extensions.go:45 Extension is starting...    {"kind": "extension", "name": "file_storage"}
2022-09-22T11:11:52.832Z    info    extensions/extensions.go:49 Extension started.  {"kind": "extension", "name": "file_storage"}
2022-09-22T11:11:52.832Z    info    pipelines/pipelines.go:74   Starting exporters...
2022-09-22T11:11:52.832Z    info    pipelines/pipelines.go:78   Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:82   Exporter started.   {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:78   Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "signalfx"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:82   Exporter started.   {"kind": "exporter", "data_type": "metrics", "name": "signalfx"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:78   Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:82   Exporter started.   {"kind": "exporter", "data_type": "logs", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:78   Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:82   Exporter started.   {"kind": "exporter", "data_type": "traces", "name": "otlp"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:78   Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "signalfx"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:82   Exporter started.   {"kind": "exporter", "data_type": "traces", "name": "signalfx"}
2022-09-22T11:11:52.833Z    info    pipelines/pipelines.go:86   Starting processors...
2022-09-22T11:11:52.834Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource/add_environment", "pipeline": "traces"}
2022-09-22T11:11:52.834Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource/add_environment", "pipeline": "traces"}
2022-09-22T11:11:52.834Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource", "pipeline": "traces"}
2022-09-22T11:11:52.834Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource", "pipeline": "traces"}
2022-09-22T11:11:52.834Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
2022-09-22T11:11:52.834Z    info    internal/resourcedetection.go:136   began detecting resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.835Z    warn    internal/resourcedetection.go:141   failed to detect resource   {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "error": "isEks() error retrieving auth configmap: failed to retrieve ConfigMap kube-system/aws-auth: configmaps \"aws-auth\" is forbidden: User \"system:serviceaccount:infra:splunk-otel\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""}
2022-09-22T11:11:52.939Z    info    internal/resourcedetection.go:150   detected resource information   {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "resource": {"cloud.account.id":"789619356883","cloud.availability_zone":"us-west-2c","cloud.platform":"aws_ec2","cloud.provider":"aws","cloud.region":"us-west-2","host.id":"i-05abc9df54abefab7","host.image.id":"ami-05842abec80265d2f","host.name":"ip-10-200-175-36.us-west-2.compute.internal","host.type":"m5n.2xlarge","os.type":"linux"}}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "batch", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "k8sattributes", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "k8sattributes", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "memory_limiter", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "memory_limiter", "pipeline": "traces"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource/add_environment", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource/add_environment", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "k8sattributes", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "k8sattributes", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "resource/add_agent_k8s", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "resource/add_agent_k8s", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "batch", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:90   Processor is starting...    {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:94   Processor started.  {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:98   Starting receivers...
2022-09-22T11:11:52.939Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:52.939Z    info    adapter/receiver.go:54  Starting stanza receiver    {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:52.940Z    info    fileconsumer/file.go:317    Resuming from previously known offset(s). 'start_at' setting is not applicable. {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer"}
2022-09-22T11:11:53.130Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:53.130Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317   {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318   {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "receiver_creator", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "receiver_creator", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "signalfx", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "signalfx", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "kubeletstats", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "kubeletstats", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-22T11:11:53.139Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z    info    static/strategy_store.go:203    No sampling strategies provided or URL is unavailable, using defaults   {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces"}
2022-09-22T11:11:53.140Z    info    smartagentreceiver/receiver.go:121  This Smart Agent monitor does not use Host/Port config fields. If either are set, they will be ignored. {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces", "monitor_type": "signalfx-forwarder"}
2022-09-22T11:11:53.141Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces"}
2022-09-22T11:11:53.141Z    info    pipelines/pipelines.go:102  Receiver is starting... {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
2022-09-22T11:11:53.141Z    info    pipelines/pipelines.go:106  Receiver started.   {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
2022-09-22T11:11:53.141Z    info    healthcheck/handler.go:129  Health Check state change   {"kind": "extension", "name": "health_check", "status": "ready"}
2022-09-22T11:11:53.141Z    info    service/collector.go:215    Starting otelcol... {"Version": "v0.57.0", "NumCPU": 8}
2022-09-22T11:11:53.141Z    info    service/collector.go:128    Everything is ready. Begin running and processing data.
2022-09-22T11:11:53.336Z    info    fileconsumer/file.go:164    Started watching file   {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/auth-dev_ed36362e-9f2b-4357-8288-9f48048ccdfb.finch-canary.1-p264t_fadf8c73-31bd-495d-bddc-d94e7cb9033c/finch-canary/0.log"}
2022-09-22T11:11:58.541Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "6.892883706s"}
2022-09-22T11:11:58.927Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:00.037Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:00.037Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:05.632Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "interval": "5.933379252s"}
2022-09-22T11:12:05.737Z    info    fileconsumer/file.go:164    Started watching file   {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/datamanagement-dev_clientupload-5b447955b9-ctqmn_f94522d9-4f73-45b7-87af-36b724e49ab7/service/2162.log"}
2022-09-22T11:12:08.374Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "3.208491575s"}
2022-09-22T11:12:09.001Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:09.002Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "5.46959699s"}
2022-09-22T11:12:10.533Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:16.568Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "9.554671545s"}
2022-09-22T11:12:18.405Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.16332784s"}
2022-09-22T11:12:19.205Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "7.377964955s"}
2022-09-22T11:12:19.237Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:20.005Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:27.730Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "interval": "10.457760927s"}
2022-09-22T11:12:28.533Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.51350341s"}
2022-09-22T11:12:30.501Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:30.584Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "14.055262332s"}
2022-09-22T11:12:32.805Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:39.048Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.97565188s"}
2022-09-22T11:12:41.253Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:42.847Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "11.189016303s"}
2022-09-22T11:12:48.353Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "21.486926859s"}
2022-09-22T11:12:50.469Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:12:52.793Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "9.865807599s"}
2022-09-22T11:12:55.028Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "6.345975272s"}
2022-09-22T11:12:55.077Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:02.190Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "17.454785105s"}
2022-09-22T11:13:04.293Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:05.555Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "5.267897057s"}
2022-09-22T11:13:09.942Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "21.713716312s"}
2022-09-22T11:13:12.229Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:12.626Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "38.458567028s"}
2022-09-22T11:13:14.737Z    info    fileconsumer/file.go:164    Started watching file   {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/datamanagement-dev_deviceintegrity-5ff7cc8f4d-fbh5n_3133b30e-bd56-4b26-92fb-48dce001955c/service/428.log"}
2022-09-22T11:13:15.825Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "10.062998176s"}
2022-09-22T11:13:21.701Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "32.507449546s"}
2022-09-22T11:13:23.136Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:23.136Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.732960584s"}
2022-09-22T11:13:29.970Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "26.603743432s"}
2022-09-22T11:13:37.753Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "37.072012422s"}
2022-09-22T11:13:39.815Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "36.104852721s"}
2022-09-22T11:13:39.877Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:46.619Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "9.750651881s"}
2022-09-22T11:13:48.640Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.140441722s"}
2022-09-22T11:13:51.870Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "22.714005922s"}
2022-09-22T11:13:54.029Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:13:59.726Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "33.075274041s"}
2022-09-22T11:14:01.372Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "12.713077492s"}
2022-09-22T11:14:02.640Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "25.813897389s"}
2022-09-22T11:14:13.357Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "23.546339274s"}
2022-09-22T11:14:13.460Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "36.893184057s"}
2022-09-22T11:14:14.584Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "39.465907256s"}
2022-09-22T11:14:15.461Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:14:23.929Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "37.226995363s"}
2022-09-22T11:14:29.032Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "31.519652929s"}
2022-09-22T11:14:30.473Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "35.896439113s"}
2022-09-22T11:14:32.613Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:14:36.904Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "27.229370369s"}
2022-09-22T11:14:39.206Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.422550359s"}
2022-09-22T11:14:45.059Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.31967519s"}
2022-09-22T11:14:47.545Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "20.971841389s"}
2022-09-22T11:14:52.838Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.914706617s"}
2022-09-22T11:15:01.196Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.255845431s"}
2022-09-22T11:15:05.553Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "37.347812839s"}
2022-09-22T11:15:06.157Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "28.250916605s"}
2022-09-22T11:15:08.453Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:15:17.089Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "24.919915587s"}
2022-09-22T11:15:17.787Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "39.640556565s"}
2022-09-22T11:15:19.205Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:15:21.435Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "44.018814291s"}
2022-09-22T11:15:28.310Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "22.617998379s"}
2022-09-22T11:15:31.334Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "15.189070057s"}
2022-09-22T11:15:34.409Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "31.965885264s"}
2022-09-22T11:15:38.714Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.204179734s"}
2022-09-22T11:15:44.729Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "20.532216889s"}
2022-09-22T11:15:47.531Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "28.624841901s"}
2022-09-22T11:15:50.928Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "36.611230157s"}
2022-09-22T11:15:58.715Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "19.386132803s"}
2022-09-22T11:16:01.732Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.468755751s"}
2022-09-22T11:16:01.919Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "19.285159386s"}
2022-09-22T11:16:10.174Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "26.255696392s"}
2022-09-22T11:16:12.390Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "16.182063727s"}
2022-09-22T11:16:16.567Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "36.67750751s"}
2022-09-22T11:16:21.205Z    error   exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/logs.go:131
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:23.102Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "40.06773469s"}
2022-09-22T11:16:23.549Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "24.192496731s"}
2022-09-22T11:16:25.257Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
2022-09-22T11:16:29.986Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "18.060101236s"}
2022-09-22T11:16:30.380Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host"    {"grpc_log": true}
2022-09-22T11:16:33.491Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.123392939s"}
2022-09-22T11:16:34.799Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "interval": "10.608712947s"}
2022-09-22T11:16:38.847Z    error   exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/logs.go:131
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:41.547Z    error   exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "max elapsed time expired Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "dropped_items": 28}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
    /builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:43.602Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "27.184918859s"}
2022-09-22T11:16:45.052Z    info    exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "interval": "2.525263981s"}
johnzheng1975 commented 2 years ago

From upper log, is this the root reason?

2022-09-22T11:17:42.356Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
  "Addr": "splunk-otel:4317",
  "ServerName": "splunk-otel:4317",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out"    {"grpc_log": true}
johnzheng1975 commented 2 years ago

$ k exec -ti -n infra splunk-otel-agent-77bkw -- curl http://splunk-otel:9943/v2/datapoint -XPOST

curl: (7) Failed to connect to splunk-otel port 9943: Connection timed out

johnzheng1975 commented 2 years ago

Sometimes: $ k exec -ti -n infra splunk-otel-agent-77bkw -- curl splunk-otel:4317 Warning: Binary output can mess up your terminal. Use "--output -" to tell Sometimes: $ k exec -ti -n infra splunk-otel-agent-77bkw -- curl http://splunk-otel:4317 curl: (7) Failed to connect to splunk-otel port 4317: Connection timed out command terminated with exit code 7

johnzheng1975 commented 2 years ago

The result is: Totally 9 nodes, only 7 nodes has forwarded the logs successfully. The other two nodes has no logs showed in splunk.

atoulme commented 2 years ago

Hello @johnzheng1975 would you please open a support case if possible? This will be a better way to discuss and dive deeper on the issues you face.

johnzheng1975 commented 2 years ago

ok, yes, submitted the ticket. BTW, I resolved this issue through increase the pods count

Not sure this is root reason or not.