Question: Validate or fail pod creation if sidecar.opentelemetry.io/inject configured collector doesn't exist

[GabrielDyck]

Component(s)

operator, collector

Describe the issue you're reporting

Hello!

I am integrating OTel Collector in our platform and I am creating a new OpenTelemetryCollector in every new deployment given a configuration provided by our users.

My goal is to launch the deployment after creating the otel collector and checking that the collector is ready to inject the sidecar on it, because I don't want to continue the deployment process if an otc-container component is missing.

Question

I realize that if the configured "sidecar.opentelemetry.io/inject" collector doesn't exist, the pod starts without the sidecar. So I am curious about how and when the collector configuration is ready to inject sidecars in new pods.

1. I would like to know if there is an endpoint to check that the operator is ready to catch the new pods and inject the configured collector or if it is unnecessary.
2. Is there any way to fail pod creation if otel collector config is not present in otel operator or if operator is not up and running?

Thank you in advance!

I've uploaded my configuration in the following section.

Configuration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: opentelemetry-operator-system/opentelemetry-operator-serving-cert
  labels:
    app.kubernetes.io/name: opentelemetry-operator
  name: opentelemetry-operator-mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /mutate-opentelemetry-io-v1alpha1-instrumentation
  failurePolicy: Fail
  name: minstrumentation.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instrumentations
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /mutate-opentelemetry-io-v1alpha1-opampbridge
  failurePolicy: Fail
  name: mopampbridge.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - opampbridges
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
  failurePolicy: Fail
  name: mopentelemetrycollector.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - opentelemetrycollectors
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /mutate-v1-pod
  failurePolicy: Ignore
  name: mpod.kb.io
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: opentelemetry-operator-system/opentelemetry-operator-serving-cert
  labels:
    app.kubernetes.io/name: opentelemetry-operator
  name: opentelemetry-operator-validating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-instrumentation
  failurePolicy: Fail
  name: vinstrumentationcreateupdate.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instrumentations
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-instrumentation
  failurePolicy: Ignore
  name: vinstrumentationdelete.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - DELETE
    resources:
    - instrumentations
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-opampbridge
  failurePolicy: Fail
  name: vopampbridgecreateupdate.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - opampbridges
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-opampbridge
  failurePolicy: Ignore
  name: vopampbridgedelete.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - DELETE
    resources:
    - opampbridges
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-opentelemetrycollector
  failurePolicy: Fail
  name: vopentelemetrycollectorcreateupdate.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - opentelemetrycollectors
  sideEffects: None
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: opentelemetry-operator-webhook-service
      namespace: opentelemetry-operator-system
      path: /validate-opentelemetry-io-v1alpha1-opentelemetrycollector
  failurePolicy: Ignore
  name: vopentelemetrycollectordelete.kb.io
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - DELETE
    resources:
    - opentelemetrycollectors
  sideEffects: None

[GabrielDyck]

I've read the code and I've found that in https://github.com/open-telemetry/opentelemetry-operator/blob/main/pkg/sidecar/podmutator.go#L81 operator allows the pod creation even if there are errors.

GIven that It reads collectors from k8s, I think that , for my use case, 'm fine assuming that if my collector is present in the Kubernetes, it is enough to continue with my deployment.

On the other hand, I was wondering if this is something that we could work to catch error cases that can lead to missing metrics. Maybe adding new annotations or operator config to indicate what the operator should do in those cases.

Let me know if I can do something from my end.