open-telemetry / opentelemetry-operator

Kubernetes Operator for OpenTelemetry Collector
Apache License 2.0
1.21k stars 440 forks source link

Operator doesn't delete components when removed from configuration #3454

Open sfc-gh-dguy opened 3 days ago

sfc-gh-dguy commented 3 days ago

Component(s)

collector, target allocator

What happened?

Description

I have an otel operator that creates otel collectors as daemonset and an otel target allocator based on an OpenTelemetryCollector CR. I removed the target allocator part from the OpenTelemetryCollector CR but the target allocator wasn't deleted. Only when I manually delete it, it's removed and is not recreated as it was removed from the config. I also deleted the whole OpenTelemetryCollector CR and both the daemonset and the target allocator weren't deleted.

Steps to Reproduce

  1. Create an OpenTelemetryCollector CR that defines collectors daemonset and atarget allocator deployment
  2. Let the operator create the components
  3. Delete the target allocator part from the CR

Expected Result

Target allocator should be deleted when removed from the CR

Actual Result

Target allocator wasn't deleted when removed from the CR

Kubernetes Version

1.29.8

Operator version

0.103.0

Collector version

0.111.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

Log output

These are the only prints I see in the operator logs:
{"level":"INFO","timestamp":"2024-11-13T16:43:17.143921523Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:43:17.362086635Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:43:17.543764697Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.3789637Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.554856277Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.761659569Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}

Additional context

No response

swiatekm commented 22 hours ago

I can reproduce the part where the target allocator resources are not deleted if you disable it. This is a more general problem with how we handle these kinds of resources, whose lifecycle is not the same as the OpenTelemetryCollector CR lifecycle.

I can't reproduce the issue where deleting the OpenTelemetryCollector CR leaves orphaned resources behind. That's actually not something the operator does directly - we simply set an owner reference, and rely on Kubernetes garbage collection. Are you able to reproduce this?