Closed johnzheng1975 closed 2 years ago
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: splunk-otel
namespace: infra
spec:
interval: 10m
url: https://signalfx.github.io/splunk-otel-collector-chart
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: splunk-otel
namespace: infra
spec:
interval: 5m # The interval at which to reconcile the Helm release
releaseName: splunk-otel
chart:
spec:
chart: splunk-otel-collector
version: 0.54.0
sourceRef:
kind: HelmRepository
name: splunk-otel
valuesFrom:
- kind: Secret
name: splunk-token
- kind: Secret
name: splunk-otel-token
values:
nameOverride: splunk-otel
clusterName: ${ENVIRONMENT}${ENVIRONMENT_SUFFIX}
environment: ${ENVIRONMENT}
extraAttributes:
fromLabels:
- key: app
tag_name: label_app
from: pod
- key: release
tag_name: label_release
from: pod
- key: type
tag_name: label_type
from: pod
custom:
- name: cluster_codename
value: ${CLUSTER_NAME}
- name: cluster_region
value: ${AWS_REGION}
- name: project
value: onecloud
splunkPlatform:
endpoint: https://${SPLUNK_HOST_NAME}:8088/services/collector
insecureSkipVerify: true
timeout: 30s
index: infra-${ENVIRONMENT}
fieldNameConvention:
renameFieldsSck: true
keepOtelConvention: false
logsEngine: otel
splunkObservability:
realm: us1
#metricsEnabled: false
cloudProvider: aws
distribution: eks
gateway:
enabled: true
replicaCount: ${SPLUNK_OTEL_GATEWAY_REPLICAS}
resources:
requests:
cpu: ${SPLUNK_OTEL_GATEWAY_REQUEST_CPU}
clusterReceiver:
resources:
limits:
cpu: ${SPLUNK_OTEL_RECEIVER_LIMIT_CPU}
memory: ${SPLUNK_OTEL_RECEIVER_LIMIT_MEMORY}
agent:
resources:
requests:
memory: ${SPLUNK_OTEL_AGNET_REQUEST_MEMORY}
limits:
memory: ${SPLUNK_OTEL_AGENT_LIMIT_MEMORY}
``
logs is:
(newenv): oc-dev-blue@PS0IMMjohn:~/tmp/logmissing$ k logs -n infra splunk-otel-agent-kvmfs
2022/09/22 11:11:52 main.go:225: Set config to [/conf/relay.yaml]
2022/09/22 11:11:52 main.go:291: Set ballast to 1351 MiB
2022/09/22 11:11:52 main.go:305: Set memory limit to 3686 MiB
2022-09-22T11:11:52.534Z info service/telemetry.go:103 Setting up own telemetry...
2022-09-22T11:11:52.535Z info service/telemetry.go:138 Serving Prometheus metrics {"address": "0.0.0.0:8889", "level": "basic"}
2022-09-22T11:11:52.535Z info memorylimiterprocessor/memorylimiter.go:113 Memory limiter configured {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "limit_mib": 3686, "spike_limit_mib": 737, "check_interval": 2}
2022-09-22T11:11:52.538Z info signalfxexporter@v0.57.2/factory.go:91 Correlation tracking enabled {"kind": "exporter", "data_type": "traces", "name": "signalfx", "endpoint": "http://splunk-otel:6060"}
2022-09-22T11:11:52.637Z info extensions/extensions.go:42 Starting extensions...
2022-09-22T11:11:52.637Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "health_check"}
2022-09-22T11:11:52.637Z info healthcheckextension@v0.57.2/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-22T11:11:52.637Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "health_check"}
2022-09-22T11:11:52.637Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "k8s_observer"}
2022-09-22T11:11:52.637Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "k8s_observer"}
2022-09-22T11:11:52.637Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "memory_ballast"}
2022-09-22T11:11:52.830Z info ballastextension/memory_ballast.go:52 Setting memory ballast {"kind": "extension", "name": "memory_ballast", "MiBs": 1351}
2022-09-22T11:11:52.832Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "memory_ballast"}
2022-09-22T11:11:52.832Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z info zpagesextension/zpagesextension.go:64 Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z info zpagesextension/zpagesextension.go:74 Registered Host's zPages {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z info zpagesextension/zpagesextension.go:86 Starting zPages extension {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2022-09-22T11:11:52.832Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "zpages"}
2022-09-22T11:11:52.832Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "file_storage"}
2022-09-22T11:11:52.832Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "file_storage"}
2022-09-22T11:11:52.832Z info pipelines/pipelines.go:74 Starting exporters...
2022-09-22T11:11:52.832Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "signalfx"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "metrics", "name": "signalfx"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "logs", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "otlp"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "signalfx"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "signalfx"}
2022-09-22T11:11:52.833Z info pipelines/pipelines.go:86 Starting processors...
2022-09-22T11:11:52.834Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource/add_environment", "pipeline": "traces"}
2022-09-22T11:11:52.834Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource/add_environment", "pipeline": "traces"}
2022-09-22T11:11:52.834Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource", "pipeline": "traces"}
2022-09-22T11:11:52.834Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource", "pipeline": "traces"}
2022-09-22T11:11:52.834Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
2022-09-22T11:11:52.834Z info internal/resourcedetection.go:136 began detecting resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.835Z warn internal/resourcedetection.go:141 failed to detect resource {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "error": "isEks() error retrieving auth configmap: failed to retrieve ConfigMap kube-system/aws-auth: configmaps \"aws-auth\" is forbidden: User \"system:serviceaccount:infra:splunk-otel\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""}
2022-09-22T11:11:52.939Z info internal/resourcedetection.go:150 detected resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "resource": {"cloud.account.id":"789619356883","cloud.availability_zone":"us-west-2c","cloud.platform":"aws_ec2","cloud.provider":"aws","cloud.region":"us-west-2","host.id":"i-05abc9df54abefab7","host.image.id":"ami-05842abec80265d2f","host.name":"ip-10-200-175-36.us-west-2.compute.internal","host.type":"m5n.2xlarge","os.type":"linux"}}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "k8sattributes", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "k8sattributes", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "memory_limiter", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "memory_limiter", "pipeline": "traces"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource/add_environment", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource/add_environment", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "k8sattributes", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "k8sattributes", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource/add_agent_k8s", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource/add_agent_k8s", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics/agent"}
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:98 Starting receivers...
2022-09-22T11:11:52.939Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:52.939Z info adapter/receiver.go:54 Starting stanza receiver {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:52.940Z info fileconsumer/file.go:317 Resuming from previously known offset(s). 'start_at' setting is not applicable. {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer"}
2022-09-22T11:11:53.130Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-09-22T11:11:53.130Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317 {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318 {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "receiver_creator", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "receiver_creator", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "signalfx", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "signalfx", "pipeline": "metrics"}
2022-09-22T11:11:53.131Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "kubeletstats", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "kubeletstats", "pipeline": "metrics"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-22T11:11:53.139Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z info static/strategy_store.go:203 No sampling strategies provided or URL is unavailable, using defaults {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-09-22T11:11:53.140Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces"}
2022-09-22T11:11:53.140Z info smartagentreceiver/receiver.go:121 This Smart Agent monitor does not use Host/Port config fields. If either are set, they will be ignored. {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces", "monitor_type": "signalfx-forwarder"}
2022-09-22T11:11:53.141Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "smartagent/signalfx-forwarder", "pipeline": "traces"}
2022-09-22T11:11:53.141Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
2022-09-22T11:11:53.141Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
2022-09-22T11:11:53.141Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
2022-09-22T11:11:53.141Z info service/collector.go:215 Starting otelcol... {"Version": "v0.57.0", "NumCPU": 8}
2022-09-22T11:11:53.141Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-09-22T11:11:53.336Z info fileconsumer/file.go:164 Started watching file {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/auth-dev_ed36362e-9f2b-4357-8288-9f48048ccdfb.finch-canary.1-p264t_fadf8c73-31bd-495d-bddc-d94e7cb9033c/finch-canary/0.log"}
2022-09-22T11:11:58.541Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "6.892883706s"}
2022-09-22T11:11:58.927Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:00.037Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:00.037Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:05.632Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "interval": "5.933379252s"}
2022-09-22T11:12:05.737Z info fileconsumer/file.go:164 Started watching file {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/datamanagement-dev_clientupload-5b447955b9-ctqmn_f94522d9-4f73-45b7-87af-36b724e49ab7/service/2162.log"}
2022-09-22T11:12:08.374Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "3.208491575s"}
2022-09-22T11:12:09.001Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:09.002Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "5.46959699s"}
2022-09-22T11:12:10.533Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:16.568Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "9.554671545s"}
2022-09-22T11:12:18.405Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.16332784s"}
2022-09-22T11:12:19.205Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "7.377964955s"}
2022-09-22T11:12:19.237Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:20.005Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:27.730Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "interval": "10.457760927s"}
2022-09-22T11:12:28.533Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.51350341s"}
2022-09-22T11:12:30.501Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:30.584Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "14.055262332s"}
2022-09-22T11:12:32.805Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:39.048Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.97565188s"}
2022-09-22T11:12:41.253Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:42.847Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "11.189016303s"}
2022-09-22T11:12:48.353Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "21.486926859s"}
2022-09-22T11:12:50.469Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:12:52.793Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "9.865807599s"}
2022-09-22T11:12:55.028Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "6.345975272s"}
2022-09-22T11:12:55.077Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:02.190Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "17.454785105s"}
2022-09-22T11:13:04.293Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:05.555Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "5.267897057s"}
2022-09-22T11:13:09.942Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "21.713716312s"}
2022-09-22T11:13:12.229Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:12.626Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "38.458567028s"}
2022-09-22T11:13:14.737Z info fileconsumer/file.go:164 Started watching file {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/datamanagement-dev_deviceintegrity-5ff7cc8f4d-fbh5n_3133b30e-bd56-4b26-92fb-48dce001955c/service/428.log"}
2022-09-22T11:13:15.825Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "10.062998176s"}
2022-09-22T11:13:21.701Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "32.507449546s"}
2022-09-22T11:13:23.136Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:23.136Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.732960584s"}
2022-09-22T11:13:29.970Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "26.603743432s"}
2022-09-22T11:13:37.753Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "37.072012422s"}
2022-09-22T11:13:39.815Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "36.104852721s"}
2022-09-22T11:13:39.877Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:46.619Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "9.750651881s"}
2022-09-22T11:13:48.640Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.140441722s"}
2022-09-22T11:13:51.870Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "22.714005922s"}
2022-09-22T11:13:54.029Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:13:59.726Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "33.075274041s"}
2022-09-22T11:14:01.372Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "12.713077492s"}
2022-09-22T11:14:02.640Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "25.813897389s"}
2022-09-22T11:14:13.357Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "23.546339274s"}
2022-09-22T11:14:13.460Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "36.893184057s"}
2022-09-22T11:14:14.584Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "39.465907256s"}
2022-09-22T11:14:15.461Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:14:23.929Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "37.226995363s"}
2022-09-22T11:14:29.032Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "31.519652929s"}
2022-09-22T11:14:30.473Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "35.896439113s"}
2022-09-22T11:14:32.613Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:14:36.904Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "27.229370369s"}
2022-09-22T11:14:39.206Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.422550359s"}
2022-09-22T11:14:45.059Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.31967519s"}
2022-09-22T11:14:47.545Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "20.971841389s"}
2022-09-22T11:14:52.838Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.914706617s"}
2022-09-22T11:15:01.196Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.255845431s"}
2022-09-22T11:15:05.553Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "37.347812839s"}
2022-09-22T11:15:06.157Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "28.250916605s"}
2022-09-22T11:15:08.453Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:15:17.089Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "24.919915587s"}
2022-09-22T11:15:17.787Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "39.640556565s"}
2022-09-22T11:15:19.205Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:15:21.435Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "44.018814291s"}
2022-09-22T11:15:28.310Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "22.617998379s"}
2022-09-22T11:15:31.334Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "15.189070057s"}
2022-09-22T11:15:34.409Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "31.965885264s"}
2022-09-22T11:15:38.714Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "23.204179734s"}
2022-09-22T11:15:44.729Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "20.532216889s"}
2022-09-22T11:15:47.531Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "28.624841901s"}
2022-09-22T11:15:50.928Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "36.611230157s"}
2022-09-22T11:15:58.715Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "19.386132803s"}
2022-09-22T11:16:01.732Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "25.468755751s"}
2022-09-22T11:16:01.919Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "19.285159386s"}
2022-09-22T11:16:10.174Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "26.255696392s"}
2022-09-22T11:16:12.390Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "16.182063727s"}
2022-09-22T11:16:16.567Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "36.67750751s"}
2022-09-22T11:16:21.205Z error exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/logs.go:131
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:23.102Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "40.06773469s"}
2022-09-22T11:16:23.549Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "24.192496731s"}
2022-09-22T11:16:25.257Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
2022-09-22T11:16:29.986Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "18.060101236s"}
2022-09-22T11:16:30.380Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host" {"grpc_log": true}
2022-09-22T11:16:33.491Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out\"", "interval": "35.123392939s"}
2022-09-22T11:16:34.799Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "interval": "10.608712947s"}
2022-09-22T11:16:38.847Z error exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/logs.go:131
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:41.547Z error exporterhelper/queued_retry_inmemory.go:115 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "max elapsed time expired Post \"http://splunk-otel:9943/v2/datapoint\": dial tcp 172.20.236.54:9943: i/o timeout (Client.Timeout exceeded while awaiting headers)", "dropped_items": 28}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry.go:199
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/queued_retry_inmemory.go:142
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.57.2/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-22T11:16:43.602Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"http://splunk-otel:9943/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "27.184918859s"}
2022-09-22T11:16:45.052Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.236.54:4317: connect: no route to host\"", "interval": "2.525263981s"}
From upper log, is this the root reason?
2022-09-22T11:17:42.356Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
"Addr": "splunk-otel:4317",
"ServerName": "splunk-otel:4317",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.20.236.54:4317: connect: connection timed out" {"grpc_log": true}
$ k exec -ti -n infra splunk-otel-agent-77bkw -- curl http://splunk-otel:9943/v2/datapoint -XPOST
curl: (7) Failed to connect to splunk-otel port 9943: Connection timed out
Sometimes: $ k exec -ti -n infra splunk-otel-agent-77bkw -- curl splunk-otel:4317 Warning: Binary output can mess up your terminal. Use "--output -" to tell Sometimes: $ k exec -ti -n infra splunk-otel-agent-77bkw -- curl http://splunk-otel:4317 curl: (7) Failed to connect to splunk-otel port 4317: Connection timed out command terminated with exit code 7
The result is: Totally 9 nodes, only 7 nodes has forwarded the logs successfully. The other two nodes has no logs showed in splunk.
Hello @johnzheng1975 would you please open a support case if possible? This will be a better way to discuss and dive deeper on the issues you face.
ok, yes, submitted the ticket. BTW, I resolved this issue through increase the pods count
Not sure this is root reason or not.
splunk-otel-agent raise error: Post http://splunk-otel:9943/v2/datapoint context deadline exceeded (Client.Timeout exceeded while awaiting headers)