search

open-telemetry / opentelemetry-helm-charts

OpenTelemetry Helm Charts
https://opentelemetry.io
Apache License 2.0
385 stars 462 forks source link

Webhook timeout when deploying collector #625

Open chris-minka opened 1 year ago

chris-minka commented 1 year ago

I have deployed the operator (chart version 0.21.4) using the default chart values into a namespace called application. The pod starts and seems healthy. When I try deploy the collector there is an error:

$ kubectl -n application apply -f kube-manifests/open-telemetry/collector.yaml
Error from server (InternalError): error when creating "kube-manifests/open-telemetry/collector.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io": failed to call webhook: Post "https://opentelemetry-operator-webhook-service.application.svc:443/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector?timeout=10s": context deadline exceeded

collector.yaml:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: application
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      googlecloud:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging, googlecloud]

The firewall allows traffic on 443. Also, a test pod does not timeout when calling the webhook URL directly:

# curl -k -X POST https://opentelemetry-operator-webhook-service.application.svc:443/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector?timeout=30s
{"response":{"uid":"","allowed":false,"status":{"metadata":{},"message":"contentType=, expected application/json","code":400}}}
povilasv commented 1 year ago

@chris-minka hey! What is weird about your error is:

Error from server (InternalError): error when creating "kube-manifests/open-telemetry/collector.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io"

the webhook is named mopentelemetrycollector.kb.io, see the weird m in the beginning of the name ?

I just did a fresh install and for me webhook is named:

opentelemetry-operator-mutating-webhook-configuration

and your example collector deployed without problems.

Any chance you have some webhook from old installation or smth like that?

Could you show us your webhook configs?

kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
chris-minka commented 1 year ago

hi @povilasv! thanks for the reply.

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io 
NAME                                                      WEBHOOKS   AGE
cert-manager-webhook                                      1          419d
gke-vpa-webhook-config                                    1          5h25m
gmp-operator.gmp-system.monitoring.googleapis.com         2          5h24m
neg-annotation.config.common-webhooks.networking.gke.io   1          2y51d
opentelemetry-operator-mutating-webhook-configuration     3          45h
pod-ready.config.common-webhooks.networking.gke.io        1          2y168d
$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration -o yaml | head -n 62
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
    meta.helm.sh/release-name: opentelemetry-operator
    meta.helm.sh/release-namespace: application
  creationTimestamp: "2023-02-01T15:30:19Z"
  generation: 2
  labels:
    app.kubernetes.io/component: webhook
    app.kubernetes.io/instance: opentelemetry-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opentelemetry-operator
    app.kubernetes.io/version: 0.67.0
    helm.sh/chart: opentelemetry-operator-0.21.4
    helm.toolkit.fluxcd.io/name: open-telemetry-operator
    helm.toolkit.fluxcd.io/namespace: application
  name: opentelemetry-operator-mutating-webhook-configuration
  resourceVersion: "745462016"
  uid: 3c8746c1-76c1-41c4-a533-bbcfca54c7ee
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: opentelemetry-operator-webhook-service
      namespace: application
      path: /mutate-opentelemetry-io-v1alpha1-instrumentation
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: minstrumentation.kb.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - opentelemetry.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instrumentations
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: opentelemetry-operator-webhook-service
      namespace: application
      path: /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: mopentelemetrycollector.kb.io
povilasv commented 1 year ago

Everything looks ok, not sure what is the issue :/

Anything in operator logs?

Could you check kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration or in general kubectl get events to see if anything points to this issue?

chris-minka commented 1 year ago

I tried to deploy the Collector again and got the same error. Here is the describe:

$ kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration
Name:         opentelemetry-operator-mutating-webhook-configuration
Namespace:    
Labels:       app.kubernetes.io/component=webhook
              app.kubernetes.io/instance=opentelemetry-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=opentelemetry-operator
              app.kubernetes.io/version=0.67.0
              helm.sh/chart=opentelemetry-operator-0.21.4
              helm.toolkit.fluxcd.io/name=open-telemetry-operator
              helm.toolkit.fluxcd.io/namespace=application
Annotations:  cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
              meta.helm.sh/release-name: opentelemetry-operator
              meta.helm.sh/release-namespace: application
API Version:  admissionregistration.k8s.io/v1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2023-02-01T15:30:19Z
  Generation:          2
  Managed Fields:
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"minstrumentation.kb.io"}:
          f:clientConfig:
            f:caBundle:
        k:{"name":"mopentelemetrycollector.kb.io"}:
          f:clientConfig:
            f:caBundle:
        k:{"name":"mpod.kb.io"}:
          f:clientConfig:
            f:caBundle:
    Manager:      cainjector
    Operation:    Update
    Time:         2023-02-01T15:30:19Z
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/inject-ca-from:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/component:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
          f:helm.toolkit.fluxcd.io/name:
          f:helm.toolkit.fluxcd.io/namespace:
      f:webhooks:
        .:
        k:{"name":"minstrumentation.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
        k:{"name":"mopentelemetrycollector.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
        k:{"name":"mpod.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
    Manager:         helm-controller
    Operation:       Update
    Time:            2023-02-01T15:30:19Z
  Resource Version:  745462016
  UID:               3c8746c1-76c1-41c4-a533-bbcfca54c7ee
Webhooks:
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-opentelemetry-io-v1alpha1-instrumentation
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            minstrumentation.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      opentelemetry.io
    API Versions:
      v1alpha1
    Operations:
      CREATE
      UPDATE
    Resources:
      instrumentations
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            mopentelemetrycollector.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      opentelemetry.io
    API Versions:
      v1alpha1
    Operations:
      CREATE
      UPDATE
    Resources:
      opentelemetrycollectors
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        opentelemetry-operator-webhook-service
      Namespace:   application
      Path:        /mutate-v1-pod
      Port:        443
  Failure Policy:  Ignore
  Match Policy:    Equivalent
  Name:            mpod.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:

    API Versions:
      v1
    Operations:
      CREATE
      UPDATE
    Resources:
      pods
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
Events:             <none>

There are a few events related to other things which are deployed, but nothing pertaining to this.

povilasv commented 1 year ago

What about logs of opentelemetry-operator-controller-manager-... ?

Also Kubernetes API Server might have more details, so try to look at it's logs

milesarmstrong commented 1 year ago

@chris-minka did you get anywhere with this? I was seeing the same timeout error (I've just disabled webhooks for now).

@povilasv it seems that mopentelemetrycollector.kb.io is coming from the chart, e.g. https://github.com/open-telemetry/opentelemetry-helm-charts/blob/56701aeb4c52a6fcdb459d3774dd91e6c67a723a/charts/opentelemetry-operator/templates/admission-webhooks/operator-webhook.yaml#L68

chris-minka commented 1 year ago

@milesarmstrong i ended up deploying the collector using its helm chart instead of using the operator. so, unfortunately i do not have a solution. if i were to revisit the issue, i would provide the logs @povilasv requested. also, since we are using a GKE Private Cluster i would review the firewall rules.

milesarmstrong commented 1 year ago

Thanks @chris-minka, we're also using GKE Private. I suspect it is the firewall rules, thanks for the pointer!

paulgrav commented 1 year ago

I posted this comment in the knative project, I think the issue is very similar: https://github.com/knative/serving/issues/13045#issuecomment-1359356226

We run our clusters on GKE. The webhook call is made from the apiserver. The webhook pod listens on 8443 whilst the service listens on 443. When making the webhook call the GKE apiserver tries to hit the webhook pod on 8443. Only ports 443 and 10250 are open between the apiserver and the GKE nodes.

RaiderAdam commented 1 year ago

I hit this issue also and digging into it, it is definitely a GKE firewall issue, but it is not a port 8443 issue.

When I set the allow to tcp:8443, the collector still would not build. When I opened all ports tcp:1-65535, collector did build.

So now I am trying to determine what actual port it needs and is getting blocked. I tried 8080 also but no luck. I'll report back when I figure it out (unless someone chimes in and knows what the port should be).

RaiderAdam commented 1 year ago

It's port 9443 for some reason (I brute forced found it by trial and error), even though the service describe says 8443. adam@AdamPC:~/opentelemetry$ gcloud compute firewall-rules create gke-opentelemetry-webhook \ --action ALLOW \ --direction INGRESS \ --source-ranges 172.16.0.16/28 \ --rules tcp:9443 \ --target-tags gke-cluster-19216119-node Creating firewall... Creating firewall...done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED gke-opentelemetry-webhook default INGRESS 1000 tcp:9443 False adam@AdamPC:~/opentelemetry$ kubectl -n opentelemetry apply -f collector.yaml opentelemetrycollector.opentelemetry.io/simplest created

RaiderAdam commented 1 year ago

There is an arg for a webhook port on the operator Args: --webhook-port=9443

If that webhook port gets changed to 443, there shouldn't be a need for a special firewall rule, correct? Since the default GKE firewall would be sufficient.

PavelPikat commented 12 months ago

I get the same error with version 0.36.0 of the opentelemetry-operator chart, and downgrading to 0.34.0 resolves it. I suspect the wrong webhook name was added in the newer version of the chart