search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.08k stars 2.38k forks source link

Client-side Throttling API calls for namespace when k8sobjectsreceiver no longer monitoring that namespace #31644

Closed kenfinnigan closed 4 months ago

kenfinnigan commented 8 months ago

Component(s)

receiver/k8sobjects

What happened?

Description

Lumigo has an operator which uses the OTeL Collector builder to construct an instance of the collector for in cluster collection and processing before transmitting traces/logs/etc to a backend.

The operator tracks which namespaces need to be monitored, and causes the OTeL Collector instance to be rebuilt when namespaces are added or removed. For each namespace being monitored, it defines the k8sobjectsreceiver with k8s objects and events for a specific set of object kinds (Pod/Deployment/etc).

When a namespace is removed from monitoring, and the Collector is restarted with any receiver config for that namespace removed, there are large numbers of errors in the Collector logs about API requests being throttled.

Steps to Reproduce

Expected Result

Collector is no longer retrieving k8s events and objects for the namespace.

Actual Result

Errors in the log for kube api calls to the no longer monitored namespace being throttled:

I0307 16:43:34.491552       1 request.go:697] Waited for 1.393232136s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/statefulsets
I0307 16:43:44.491739       1 request.go:697] Waited for 1.391941742s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/replicasets
I0307 16:43:54.491976       1 request.go:697] Waited for 1.394410791s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/daemonsets
I0307 16:44:04.492993       1 request.go:697] Waited for 1.385453868s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/api/v1/namespaces/node2/events?fieldSelector=involvedObject.apiVersion%3Dapps%2Fv1%2CinvolvedObject.kind%3DStatefulSet%2CinvolvedObject.uid%21%3D%27%27%2CinvolvedObject.uid%21%3D

Collector version

v0.89.0

Environment information

Environment

OS: Alpine

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        auth:
          authenticator: lumigoauth/server
        include_metadata: true # Needed by `headers_setter/lumigo`
  k8sobjects/objects_ns_node1:
    auth_type: serviceAccount
    objects:
    - name: pods
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: daemonsets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: deployments
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: replicasets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: statefulsets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: cronjobs
      group: batch
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: jobs
      group: batch
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: pods
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: daemonsets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: deployments
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: replicasets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: statefulsets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: cronjobs
      group: batch
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: jobs
      group: batch
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
  k8sobjects/events_ns_node1:
    auth_type: serviceAccount
    objects:
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=v1,involvedObject.kind=Pod,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=DaemonSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=Deployment,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=ReplicaSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=StatefulSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=CronJob,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=Job,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=v1,involvedObject.kind=Pod,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=DaemonSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=Deployment,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=ReplicaSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=StatefulSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=CronJob,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=Job,involvedObject.uid!='',involvedObject.uid!=

extensions:
  health_check:
  headers_setter/lumigo:
    headers:
    - key: authorization
      from_context: Authorization
      action: upsert

exporters:
  otlphttp/lumigo:
    endpoint: https://ga-otlp.lumigo-tracer-edge.golumigo.com
    auth:
      authenticator: headers_setter/lumigo
  logging:
    verbosity: detailed
    sampling_initial: 1
    sampling_thereafter: 1
  otlphttp/lumigo_ns_node1:
    endpoint: $LUMIGO_ENDPOINT
    auth:
      authenticator: lumigoauth/ns_node1

processors:
  transform/set_k8s_objects_scope:
    log_statements:
    - context: scope
      statements:
      - set(name, "lumigo-operator.k8s-objects")
      - set(version, "28")
  transform/set_k8s_events_scope:
    log_statements:
    - context: scope
      statements:
      - set(name, "lumigo-operator.k8s-events")
      - set(version, "28")
  batch/k8s_objects_ns_node1:
    send_batch_size: 100
    timeout: 1s
  batch/k8s_events_ns_node1:
    send_batch_size: 100
    timeout: 1s

service:
  telemetry:
    logs:
      level: debug
  extensions:
  - headers_setter/lumigo
  - health_check
  - lumigoauth/server
  - lumigoauth/ns_node1
  pipelines:
    traces:
      receivers:
      - otlp
      exporters:
      - otlphttp/lumigo
    logs/k8s_objects_ns_node1:
      receivers:
      - k8sobjects/objects_ns_node1
      processors:
      - transform/set_k8s_objects_scope
      - batch/k8s_objects_ns_node1
      exporters:
      - otlphttp/lumigo_ns_node1
    logs/k8s_events_ns_node1:
      receivers:
      - k8sobjects/events_ns_node1
      processors:
      - transform/set_k8s_events_scope
      - batch/k8s_events_ns_node1
      exporters:
      - logging
      - otlphttp/lumigo_ns_node1

Log output

2024-03-07T16:37:00.682Z    info    service@v0.89.0/service.go:143  Starting lumigo-collector...    {"Version": "0.0.1", "NumCPU": 2}
2024-03-07T16:37:00.682Z    info    extensions/extensions.go:34 Starting extensions...
2024-03-07T16:37:00.682Z    info    extensions/extensions.go:37 Extension is starting...    {"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:37:00.682Z    info    extensions/extensions.go:45 Extension started.  {"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:37:00.682Z    info    extensions/extensions.go:37 Extension is starting...    {"kind": "extension", "name": "health_check"}
2024-03-07T16:37:00.682Z    info    healthcheckextension@v0.89.0/healthcheckextension.go:35 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-03-07T16:37:00.682Z    warn    internal@v0.89.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:37:00.682Z    info    extensions/extensions.go:45 Extension started.  {"kind": "extension", "name": "health_check"}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs"}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.705Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/objects_ns_node2", "data_type": "logs"}
2024-03-07T16:37:00.725Z    warn    internal@v0.89.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:37:00.725Z    info    otlpreceiver@v0.89.0/otlp.go:101    Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2024-03-07T16:37:00.725Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:37:00.725Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:37:00.726Z    info    healthcheck/handler.go:132  Health Check state change   {"kind": "extension", "name": "health_check", "status": "ready"}
2024-03-07T16:37:00.726Z    info    service@v0.89.0/service.go:169  Everything is ready. Begin running and processing data.

Reloading configurations
2024-03-07T16:41:34.890Z    info    otelcol@v0.89.0/collector.go:258    Received signal from OS {"signal": "hangup"}
2024-03-07T16:41:34.890Z    warn    otelcol@v0.89.0/collector.go:199    Config updated, restart service
2024-03-07T16:41:34.890Z    info    service@v0.89.0/service.go:178  Starting shutdown...
2024-03-07T16:41:34.890Z    info    healthcheck/handler.go:132  Health Check state change   {"kind": "extension", "name": "health_check", "status": "unavailable"}
2024-03-07T16:41:34.891Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:82   Object Receiver stopped {"kind": "receiver", "name": "k8sobjects/objects_ns_node2", "data_type": "logs"}
2024-03-07T16:41:34.891Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:82   Object Receiver stopped {"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs"}
2024-03-07T16:41:34.892Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:82   Object Receiver stopped {"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:34.892Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:82   Object Receiver stopped {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:34.894Z    info    extensions/extensions.go:52 Stopping extensions...
2024-03-07T16:41:34.895Z    info    service@v0.89.0/service.go:192  Shutdown complete.
2024-03-07T16:41:35.016Z    info    service@v0.89.0/telemetry.go:85 Setting up own telemetry...
2024-03-07T16:41:35.016Z    info    service@v0.89.0/telemetry.go:202    Serving Prometheus metrics  {"address": ":8888", "level": "Basic"}
2024-03-07T16:41:35.016Z    debug   extension@v0.89.0/extension.go:162  Beta component. May change in the future.   {"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.016Z    debug   extension@v0.89.0/extension.go:162  Stable component.   {"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.016Z    debug   extension@v0.89.0/extension.go:162  Stable component.   {"kind": "extension", "name": "lumigoauth/ns_node1"}
2024-03-07T16:41:35.016Z    debug   exporter@v0.89.0/exporter.go:273    Beta component. May change in the future.   {"kind": "exporter", "data_type": "logs", "name": "otlphttp/lumigo_ns_node1"}
2024-03-07T16:41:35.018Z    debug   processor@v0.89.0/processor.go:287  Stable component.   {"kind": "processor", "name": "batch/k8s_objects_ns_node1", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.018Z    debug   processor@v0.89.0/processor.go:287  Alpha component. May change in the future.  {"kind": "processor", "name": "transform/inject_operator_details_into_resource", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.018Z    debug   processor@v0.89.0/processor.go:287  Stable component.   {"kind": "processor", "name": "batch/k8s_events_ns_node1", "pipeline": "logs/k8s_events_ns_node1"}
2024-03-07T16:41:35.018Z    debug   processor@v0.89.0/processor.go:287  Alpha component. May change in the future.  {"kind": "processor", "name": "transform/set_k8s_objects_scope", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.019Z    debug   exporter@v0.89.0/exporter.go:273    Stable component.   {"kind": "exporter", "data_type": "traces", "name": "otlphttp/lumigo"}
2024-03-07T16:41:35.020Z    debug   receiver@v0.89.0/receiver.go:294    Stable component.   {"kind": "receiver", "name": "otlp", "data_type": "traces"}
2024-03-07T16:41:35.020Z    debug   receiver@v0.89.0/receiver.go:294    Beta component. May change in the future.   {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.020Z    debug   receiver@v0.89.0/receiver.go:294    Beta component. May change in the future.   {"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.020Z    info    service@v0.89.0/service.go:143  Starting lumigo-collector...    {"Version": "0.0.1", "NumCPU": 2}
2024-03-07T16:41:35.020Z    info    extensions/extensions.go:34 Starting extensions...
2024-03-07T16:41:35.021Z    info    extensions/extensions.go:37 Extension is starting...    {"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.021Z    info    extensions/extensions.go:45 Extension started.  {"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.021Z    info    extensions/extensions.go:37 Extension is starting...    {"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.021Z    info    healthcheckextension@v0.89.0/healthcheckextension.go:35 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-03-07T16:41:35.021Z    warn    internal@v0.89.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:41:35.021Z    info    extensions/extensions.go:45 Extension started.  {"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.034Z    info    otlpreceiver@v0.89.0/otlp.go:101    Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "/v1, Resource=pods", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=daemonsets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=deployments", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=replicasets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=statefulsets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "batch/v1, Resource=cronjobs", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "batch/v1, Resource=jobs", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "/v1, Resource=pods", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=daemonsets", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:93   Started collecting  {"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=deployments", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z    info    k8sobjectsreceiver@v0.89.0/receiver.go:73   Object Receiver started {"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.035Z    info    healthcheck/handler.go:132  Health Check state change   {"kind": "extension", "name": "health_check", "status": "ready"}
2024-03-07T16:41:35.035Z    info    service@v0.89.0/service.go:169  Everything is ready. Begin running and processing data.

Additional context

No response

github-actions[bot] commented 8 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth commented 8 months ago

I suspect the setup/teardown of the watches isnt quite right.

kenfinnigan commented 8 months ago

That's what I was thinking as well, but not sure how to debug it further to narrow down the issue.

Any suggestions @TylerHelmuth? (Bearing in mind I'm reasonably new to Golang)

TylerHelmuth commented 8 months ago

Hopefully it is somewhere in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sobjectsreceiver/receiver.go

github-actions[bot] commented 6 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 4 months ago

This issue has been closed as inactive because it has been stale for 120 days with no activity.