search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.02k stars 2.33k forks source link

k8sclusterreceiver: crash on startup #2288

Closed DanTulovsky closed 3 years ago

DanTulovsky commented 3 years ago

Describe the bug Crash on startup after adding collector:

      k8s_cluster:
        auth_type: "serviceAccount"
        collection_interval: 10s
        node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable]

Steps to reproduce Start binary with below config.

What did you expect to see? Working binary.

What did you see instead? Crash looping.

What version did you use? Version: v0.19.0

What config did you use? Config:

  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      jaeger:
        protocols:
          grpc:
          thrift_http:
          thrift_compact:
          thrift_binary:
        # remote_sampling:
          # strategy_file: "/etc/strategies.json"
          # Doesn't work
          # fetch_endpoint: "jaeger-collector.observability:5778"
      zipkin:
      # causes this crash
      k8s_cluster:
        auth_type: "serviceAccount"
        collection_interval: 10s
        node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable]
    processors:
      batch:
      memory_limiter:
        # Same as --mem-ballast-size-mib CLI argument
        ballast_size_mib: 683
        # 80% of maximum memory up to 2G
        limit_mib: 1500
        # 25% of limit up to 2G
        spike_limit_mib: 512
        check_interval: 5s
    extensions:
      health_check: {}
      zpages: {}
    exporters:
      otlp:
        endpoint: "ingest.lightstep.com:443"
        headers: redacted
      jaeger_thrift:
        url: "http://jaeger-collector.observability:14268/api/traces"
      logging:
        loglevel: info
    service:
      extensions: [health_check, zpages]
      pipelines:
        traces/1:
          receivers: [otlp, zipkin, jaeger, k8s_cluster]
          processors: [memory_limiter, batch]
          exporters: [logging, jaeger_thrift, otlp]

Environment OS: Digital Ocean Kubernetes

Additional context

2021-02-06T15:11:54.106Z    info    service/service.go:411  Starting OpenTelemetry Contrib Collector... {"Version": "v0.19.0", "GitHash": "63b2f339", "NumCPU": 2}
2021-02-06T15:11:54.143Z    info    service/service.go:592  Using memory ballast    {"MiBs": 683}
2021-02-06T15:11:54.145Z    info    service/service.go:255  Setting up own telemetry...
2021-02-06T15:11:54.148Z    info    service/telemetry.go:102    Serving Prometheus metrics  {"address": ":8888", "level": 0, "service.instance.id": "90419066-cf7e-4ac7-a5bd-20d926b04d46"}
2021-02-06T15:11:54.150Z    info    service/service.go:292  Loading configuration...
2021-02-06T15:11:54.152Z    info    service/service.go:303  Applying configuration...
2021-02-06T15:11:54.153Z    info    service/service.go:324  Starting extensions...
2021-02-06T15:11:54.153Z    info    builder/extensions_builder.go:53    Extension is starting...    {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:11:54.153Z    info    zpagesextension/zpagesextension.go:42   Register Host's zPages  {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:11:54.161Z    info    zpagesextension/zpagesextension.go:55   Starting zPages extension   {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages", "config": {"TypeVal":"zpages","NameVal":"zpages","Endpoint":"localhost:55679"}}
2021-02-06T15:11:54.162Z    info    builder/extensions_builder.go:59    Extension started.  {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:11:54.162Z    info    builder/extensions_builder.go:53    Extension is starting...    {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-02-06T15:11:54.162Z    info    healthcheckextension/healthcheckextension.go:40 Starting health_check extension {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "config": {"TypeVal":"health_check","NameVal":"health_check","Port":13133}}
2021-02-06T15:11:54.162Z    info    builder/extensions_builder.go:59    Extension started.  {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-02-06T15:11:54.163Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "jaeger_thrift"}
2021-02-06T15:11:54.163Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "logging"}
2021-02-06T15:11:54.164Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "otlp"}
2021-02-06T15:11:54.164Z    info    service/service.go:339  Starting exporters...
2021-02-06T15:11:54.164Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "jaeger_thrift", "component_name": "jaeger_thrift"}
2021-02-06T15:11:54.164Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "jaeger_thrift", "component_name": "jaeger_thrift"}
2021-02-06T15:11:54.165Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
2021-02-06T15:11:54.165Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
2021-02-06T15:11:54.165Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.165Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.166Z    info    memorylimiter/memorylimiter.go:108  Memory limiter configured   {"component_kind": "processor", "component_type": "memory_limiter", "component_name": "memory_limiter", "limit_mib": 1572864000, "spike_limit_mib": 536870912, "check_interval": 5}
2021-02-06T15:11:54.166Z    info    builder/pipelines_builder.go:207    Pipeline is enabled.    {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:11:54.166Z    info    builder/pipelines_builder.go:207    Pipeline is enabled.    {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:11:54.166Z    info    service/service.go:352  Starting processors...
2021-02-06T15:11:54.166Z    info    builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:11:54.166Z    info    builder/pipelines_builder.go:61 Pipeline is started.    {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:11:54.166Z    info    builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:11:54.167Z    info    builder/pipelines_builder.go:61 Pipeline is started.    {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:11:54.167Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin", "datatype": "traces"}
2021-02-06T15:11:54.169Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster", "datatype": "metrics"}
2021-02-06T15:11:54.170Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp", "datatype": "traces"}
2021-02-06T15:11:54.170Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger", "datatype": "traces"}
2021-02-06T15:11:54.171Z    info    service/service.go:364  Starting receivers...
2021-02-06T15:11:54.171Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin"}
2021-02-06T15:11:54.171Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin"}
2021-02-06T15:11:54.171Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:11:54.171Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:11:54.172Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.172Z    info    otlpreceiver/otlp.go:93 Starting GRPC server on endpoint 0.0.0.0:4317   {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.172Z    info    k8sclusterreceiver/receiver.go:59   Starting shared informers and wait for initial cache sync.  {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:11:54.172Z    info    otlpreceiver/otlp.go:130    Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.172Z    info    otlpreceiver/otlp.go:93 Starting GRPC server on endpoint 0.0.0.0:55680  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.173Z    info    otlpreceiver/otlp.go:108    Starting HTTP server on endpoint 0.0.0.0:55681  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.173Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:11:54.173Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:11:54.174Z    info    static/strategy_store.go:203    No sampling strategies provided or URL is unavailable, using defaults   {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:11:54.174Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:11:54.174Z    info    healthcheck/handler.go:128  Health Check state change   {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "status": "ready"}
2021-02-06T15:11:54.174Z    info    service/service.go:267  Everything is ready. Begin running and processing data.
2021-02-06T15:11:54.483Z    info    k8sclusterreceiver/receiver.go:75   Completed syncing shared informer caches.   {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
E0206 15:11:54.584199       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 185 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3098080, 0x5b49d90)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:74 +0x92
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:48 +0x82
panic(0x3098080, 0x5b49d90)
    /usr/local/go/src/runtime/panic.go:969 +0x166
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.getMetricsForJob(0xc02cbb9160, 0x3694100, 0xc02cbb8d00, 0xc02cf98ef8)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/jobs.go:73 +0x119
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.(*DataCollector).SyncMetrics(0xc02b3d5bf0, 0x3694100, 0xc02cbb9160)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/collector.go:144 +0x6e0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*resourceWatcher).onAdd(0xc000528230, 0x3694100, 0xc02cbb9160)
    /home/circleci/project/receiver/k8sclusterreceiver/watcher.go:128 +0x72
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:777 +0x129
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc02b3c3760)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc02b56bf60, 0x3dc54e0, 0xc00067a240, 0x2fd7601, 0xc00039c9c0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc02b3c3760, 0x3b9aca00, 0x0, 0x1, 0xc00039c9c0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc02b3f4a80)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000529020, 0xc00067e270)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2a21519]

goroutine 185 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:55 +0x105
panic(0x3098080, 0x5b49d90)
    /usr/local/go/src/runtime/panic.go:969 +0x166
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.getMetricsForJob(0xc02cbb9160, 0x3694100, 0xc02cbb8d00, 0xc02cf98ef8)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/jobs.go:73 +0x119
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.(*DataCollector).SyncMetrics(0xc02b3d5bf0, 0x3694100, 0xc02cbb9160)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/collector.go:144 +0x6e0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*resourceWatcher).onAdd(0xc000528230, 0x3694100, 0xc02cbb9160)
    /home/circleci/project/receiver/k8sclusterreceiver/watcher.go:128 +0x72
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:777 +0x129
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc02b3c3760)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc02b583f60, 0x3dc54e0, 0xc00067a240, 0x2fd7601, 0xc00039c9c0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc02b3c3760, 0x3b9aca00, 0x0, 0x1, 0xc00039c9c0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc02b3f4a80)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000529020, 0xc00067e270)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
2021-02-06T15:14:43.118Z    info    service/service.go:411  Starting OpenTelemetry Contrib Collector... {"Version": "v0.19.0", "GitHash": "63b2f339", "NumCPU": 2}
2021-02-06T15:14:43.151Z    info    service/service.go:592  Using memory ballast    {"MiBs": 683}
2021-02-06T15:14:43.156Z    info    service/service.go:255  Setting up own telemetry...
2021-02-06T15:14:43.174Z    info    service/telemetry.go:102    Serving Prometheus metrics  {"address": ":8888", "level": 0, "service.instance.id": "fd9d1acb-4369-4967-a4ed-0fbf7188dd99"}
2021-02-06T15:14:43.176Z    info    service/service.go:292  Loading configuration...
2021-02-06T15:14:43.177Z    info    service/service.go:303  Applying configuration...
2021-02-06T15:14:43.177Z    info    service/service.go:324  Starting extensions...
2021-02-06T15:14:43.177Z    info    builder/extensions_builder.go:53    Extension is starting...    {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:14:43.177Z    info    zpagesextension/zpagesextension.go:42   Register Host's zPages  {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:14:43.186Z    info    zpagesextension/zpagesextension.go:55   Starting zPages extension   {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages", "config": {"TypeVal":"zpages","NameVal":"zpages","Endpoint":"localhost:55679"}}
2021-02-06T15:14:43.186Z    info    builder/extensions_builder.go:59    Extension started.  {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-02-06T15:14:43.186Z    info    builder/extensions_builder.go:53    Extension is starting...    {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-02-06T15:14:43.186Z    info    healthcheckextension/healthcheckextension.go:40 Starting health_check extension {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "config": {"TypeVal":"health_check","NameVal":"health_check","Port":13133}}
2021-02-06T15:14:43.186Z    info    builder/extensions_builder.go:59    Extension started.  {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-02-06T15:14:43.187Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "logging"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "otlp"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:306    Exporter is enabled.    {"component_kind": "exporter", "exporter": "jaeger_thrift"}
2021-02-06T15:14:43.188Z    info    service/service.go:339  Starting exporters...
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "jaeger_thrift", "component_name": "jaeger_thrift"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "jaeger_thrift", "component_name": "jaeger_thrift"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
2021-02-06T15:14:43.188Z    info    builder/exporters_builder.go:97 Exporter started.   {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
2021-02-06T15:14:43.188Z    info    memorylimiter/memorylimiter.go:108  Memory limiter configured   {"component_kind": "processor", "component_type": "memory_limiter", "component_name": "memory_limiter", "limit_mib": 1572864000, "spike_limit_mib": 536870912, "check_interval": 5}
2021-02-06T15:14:43.188Z    info    builder/pipelines_builder.go:207    Pipeline is enabled.    {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:14:43.188Z    info    builder/pipelines_builder.go:207    Pipeline is enabled.    {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:14:43.188Z    info    service/service.go:352  Starting processors...
2021-02-06T15:14:43.188Z    info    builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:14:43.188Z    info    builder/pipelines_builder.go:61 Pipeline is started.    {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-02-06T15:14:43.189Z    info    builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:14:43.189Z    info    builder/pipelines_builder.go:61 Pipeline is started.    {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-02-06T15:14:43.189Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger", "datatype": "traces"}
2021-02-06T15:14:43.189Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin", "datatype": "traces"}
2021-02-06T15:14:43.192Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster", "datatype": "metrics"}
2021-02-06T15:14:43.192Z    info    builder/receivers_builder.go:235    Receiver is enabled.    {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp", "datatype": "traces"}
2021-02-06T15:14:43.192Z    info    service/service.go:364  Starting receivers...
2021-02-06T15:14:43.192Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    otlpreceiver/otlp.go:93 Starting GRPC server on endpoint 0.0.0.0:4317   {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    otlpreceiver/otlp.go:130    Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    otlpreceiver/otlp.go:93 Starting GRPC server on endpoint 0.0.0.0:55680  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    otlpreceiver/otlp.go:108    Starting HTTP server on endpoint 0.0.0.0:55681  {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "otlp", "component_name": "otlp"}
2021-02-06T15:14:43.193Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:14:43.193Z    info    static/strategy_store.go:203    No sampling strategies provided or URL is unavailable, using defaults   {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:14:43.193Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-02-06T15:14:43.194Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin"}
2021-02-06T15:14:43.194Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin"}
2021-02-06T15:14:43.194Z    info    builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:14:43.194Z    info    builder/receivers_builder.go:75 Receiver started.   {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:14:43.194Z    info    healthcheck/handler.go:128  Health Check state change   {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "status": "ready"}
2021-02-06T15:14:43.194Z    info    service/service.go:267  Everything is ready. Begin running and processing data.
2021-02-06T15:14:43.194Z    info    k8sclusterreceiver/receiver.go:59   Starting shared informers and wait for initial cache sync.  {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
2021-02-06T15:14:43.417Z    info    k8sclusterreceiver/receiver.go:75   Completed syncing shared informer caches.   {"component_kind": "receiver", "component_type": "k8s_cluster", "component_name": "k8s_cluster"}
E0206 15:14:43.517610       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 269 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3098080, 0x5b49d90)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:74 +0x92
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:48 +0x82
panic(0x3098080, 0x5b49d90)
    /usr/local/go/src/runtime/panic.go:969 +0x166
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.getMetricsForJob(0xc02bb54d60, 0x3694100, 0xc02bb54900, 0xc02caa9dc8)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/jobs.go:73 +0x119
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.(*DataCollector).SyncMetrics(0xc0005ff0b0, 0x3694100, 0xc02bb54d60)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/collector.go:144 +0x6e0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*resourceWatcher).onAdd(0xc0004fc230, 0x3694100, 0xc02bb54d60)
    /home/circleci/project/receiver/k8sclusterreceiver/watcher.go:128 +0x72
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:777 +0x129
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00073a760)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc02b5bef60, 0x3dc54e0, 0xc02b5c83c0, 0x2fd7601, 0xc02b5b67e0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00073a760, 0x3b9aca00, 0x0, 0x1, 0xc02b5b67e0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000696500)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004fca00, 0xc02b5ac970)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2a21519]

goroutine 269 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:55 +0x105
panic(0x3098080, 0x5b49d90)
    /usr/local/go/src/runtime/panic.go:969 +0x166
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.getMetricsForJob(0xc02bb54d60, 0x3694100, 0xc02bb54900, 0xc02caa9dc8)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/jobs.go:73 +0x119
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver/collection.(*DataCollector).SyncMetrics(0xc0005ff0b0, 0x3694100, 0xc02bb54d60)
    /home/circleci/project/receiver/k8sclusterreceiver/collection/collector.go:144 +0x6e0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*resourceWatcher).onAdd(0xc0004fc230, 0x3694100, 0xc02bb54d60)
    /home/circleci/project/receiver/k8sclusterreceiver/watcher.go:128 +0x72
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:777 +0x129
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00073a760)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc02b555f60, 0x3dc54e0, 0xc02b5c83c0, 0x2fd7601, 0xc02b5b67e0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00073a760, 0x3b9aca00, 0x0, 0x1, 0xc02b5b67e0)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000696500)
    /home/circleci/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004fca00, 0xc02b5ac970)
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/circleci/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
flands commented 3 years ago

@jrcamp

jrcamp commented 3 years ago

@asuresh4 ptal