Open chris-minka opened 1 year ago
@chris-minka hey! What is weird about your error is:
Error from server (InternalError): error when creating "kube-manifests/open-telemetry/collector.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io"
the webhook is named mopentelemetrycollector.kb.io
, see the weird m
in the beginning of the name ?
I just did a fresh install and for me webhook is named:
opentelemetry-operator-mutating-webhook-configuration
and your example collector deployed without problems.
Any chance you have some webhook from old installation or smth like that?
Could you show us your webhook configs?
kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
hi @povilasv! thanks for the reply.
$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
NAME WEBHOOKS AGE
cert-manager-webhook 1 419d
gke-vpa-webhook-config 1 5h25m
gmp-operator.gmp-system.monitoring.googleapis.com 2 5h24m
neg-annotation.config.common-webhooks.networking.gke.io 1 2y51d
opentelemetry-operator-mutating-webhook-configuration 3 45h
pod-ready.config.common-webhooks.networking.gke.io 1 2y168d
$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration -o yaml | head -n 62
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
meta.helm.sh/release-name: opentelemetry-operator
meta.helm.sh/release-namespace: application
creationTimestamp: "2023-02-01T15:30:19Z"
generation: 2
labels:
app.kubernetes.io/component: webhook
app.kubernetes.io/instance: opentelemetry-operator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: opentelemetry-operator
app.kubernetes.io/version: 0.67.0
helm.sh/chart: opentelemetry-operator-0.21.4
helm.toolkit.fluxcd.io/name: open-telemetry-operator
helm.toolkit.fluxcd.io/namespace: application
name: opentelemetry-operator-mutating-webhook-configuration
resourceVersion: "745462016"
uid: 3c8746c1-76c1-41c4-a533-bbcfca54c7ee
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
service:
name: opentelemetry-operator-webhook-service
namespace: application
path: /mutate-opentelemetry-io-v1alpha1-instrumentation
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: minstrumentation.kb.io
namespaceSelector: {}
objectSelector: {}
reinvocationPolicy: Never
rules:
- apiGroups:
- opentelemetry.io
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- instrumentations
scope: '*'
sideEffects: None
timeoutSeconds: 10
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
service:
name: opentelemetry-operator-webhook-service
namespace: application
path: /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: mopentelemetrycollector.kb.io
Everything looks ok, not sure what is the issue :/
Anything in operator logs?
Could you check kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration
or in general kubectl get events
to see if anything points to this issue?
I tried to deploy the Collector again and got the same error. Here is the describe:
$ kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io opentelemetry-operator-mutating-webhook-configuration
Name: opentelemetry-operator-mutating-webhook-configuration
Namespace:
Labels: app.kubernetes.io/component=webhook
app.kubernetes.io/instance=opentelemetry-operator
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=opentelemetry-operator
app.kubernetes.io/version=0.67.0
helm.sh/chart=opentelemetry-operator-0.21.4
helm.toolkit.fluxcd.io/name=open-telemetry-operator
helm.toolkit.fluxcd.io/namespace=application
Annotations: cert-manager.io/inject-ca-from: application/opentelemetry-operator-serving-cert
meta.helm.sh/release-name: opentelemetry-operator
meta.helm.sh/release-namespace: application
API Version: admissionregistration.k8s.io/v1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2023-02-01T15:30:19Z
Generation: 2
Managed Fields:
API Version: admissionregistration.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:webhooks:
k:{"name":"minstrumentation.kb.io"}:
f:clientConfig:
f:caBundle:
k:{"name":"mopentelemetrycollector.kb.io"}:
f:clientConfig:
f:caBundle:
k:{"name":"mpod.kb.io"}:
f:clientConfig:
f:caBundle:
Manager: cainjector
Operation: Update
Time: 2023-02-01T15:30:19Z
API Version: admissionregistration.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/inject-ca-from:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/component:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/managed-by:
f:app.kubernetes.io/name:
f:app.kubernetes.io/version:
f:helm.sh/chart:
f:helm.toolkit.fluxcd.io/name:
f:helm.toolkit.fluxcd.io/namespace:
f:webhooks:
.:
k:{"name":"minstrumentation.kb.io"}:
.:
f:admissionReviewVersions:
f:clientConfig:
.:
f:service:
.:
f:name:
f:namespace:
f:path:
f:port:
f:failurePolicy:
f:matchPolicy:
f:name:
f:namespaceSelector:
f:objectSelector:
f:reinvocationPolicy:
f:rules:
f:sideEffects:
f:timeoutSeconds:
k:{"name":"mopentelemetrycollector.kb.io"}:
.:
f:admissionReviewVersions:
f:clientConfig:
.:
f:service:
.:
f:name:
f:namespace:
f:path:
f:port:
f:failurePolicy:
f:matchPolicy:
f:name:
f:namespaceSelector:
f:objectSelector:
f:reinvocationPolicy:
f:rules:
f:sideEffects:
f:timeoutSeconds:
k:{"name":"mpod.kb.io"}:
.:
f:admissionReviewVersions:
f:clientConfig:
.:
f:service:
.:
f:name:
f:namespace:
f:path:
f:port:
f:failurePolicy:
f:matchPolicy:
f:name:
f:namespaceSelector:
f:objectSelector:
f:reinvocationPolicy:
f:rules:
f:sideEffects:
f:timeoutSeconds:
Manager: helm-controller
Operation: Update
Time: 2023-02-01T15:30:19Z
Resource Version: 745462016
UID: 3c8746c1-76c1-41c4-a533-bbcfca54c7ee
Webhooks:
Admission Review Versions:
v1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Service:
Name: opentelemetry-operator-webhook-service
Namespace: application
Path: /mutate-opentelemetry-io-v1alpha1-instrumentation
Port: 443
Failure Policy: Fail
Match Policy: Equivalent
Name: minstrumentation.kb.io
Namespace Selector:
Object Selector:
Reinvocation Policy: Never
Rules:
API Groups:
opentelemetry.io
API Versions:
v1alpha1
Operations:
CREATE
UPDATE
Resources:
instrumentations
Scope: *
Side Effects: None
Timeout Seconds: 10
Admission Review Versions:
v1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Service:
Name: opentelemetry-operator-webhook-service
Namespace: application
Path: /mutate-opentelemetry-io-v1alpha1-opentelemetrycollector
Port: 443
Failure Policy: Fail
Match Policy: Equivalent
Name: mopentelemetrycollector.kb.io
Namespace Selector:
Object Selector:
Reinvocation Policy: Never
Rules:
API Groups:
opentelemetry.io
API Versions:
v1alpha1
Operations:
CREATE
UPDATE
Resources:
opentelemetrycollectors
Scope: *
Side Effects: None
Timeout Seconds: 10
Admission Review Versions:
v1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURlVENDQW1HZ0F3SUJBZ0lRWWZIR1A0ZGVleS9lRTVnVStKUThBakFOQmdrcWhraUc5dzBCQVFzRkFEQWgKTVI4d0hRWURWUVFMRXhadmNHVnVkR1ZzWlcxbGRISjVMVzl3WlhKaGRHOXlNQjRYRFRJek1ERXlOekU1TXpBeQpObG9YRFRJek1EUXlOekU1TXpBeU5sb3dJVEVmTUIwR0ExVUVDeE1XYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5CllYUnZjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNWE14V1dNQ2FLZzBTZWQKU0RTbVBqWkUzVm1nL0NTczA5YkVDUlZha25peWVQd0lKNXVadVhQSUp1MTRQRmxwbmdOQlovV0w5ZHRzL1VibgpiUXA5N0c4d1lkWEIvZjVKSTBJMk9KMlIyZGFNVWkvdXY2b094cjZRSGdoV3dPelhvWjcxVENNNHFhWHpveGxYCjlqTzUyN01Ma29uNkhVaGdBOGY0RjJETSs1VnJFMnp4Z1dxdHNySGpjWEZlblZITEUwQ1hoSDU1UXNoZWEvcVEKanhQNTFDbWx4WDJaU2QxbmkycUg2UXI5SjBJREJjRzJRNTdGTGprTEtJWEdjaE5tZmYzczhyeU11aEtzNXBsbApYcGVJUVdXYzBmbnpDY2FLTGI1TGd6SGVuMzVTdmlDVzRmR1FVUUJNeXlkV0R0MGhRZmwvV2orM3hRa1RwY012Ck1CRVpPL0VDQXdFQUFhT0JyRENCcVRBT0JnTlZIUThCQWY4RUJBTUNCYUF3REFZRFZSMFRBUUgvQkFJd0FEQ0IKaUFZRFZSMFJCSUdBTUg2Q05tOXdaVzUwWld4bGJXVjBjbmt0YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMgphV05sTG1Gd2NHeHBZMkYwYVc5dUxuTjJZNEpFYjNCbGJuUmxiR1Z0WlhSeWVTMXZjR1Z5WVhSdmNpMTNaV0pvCmIyOXJMWE5sY25acFkyVXVZWEJ3YkdsallYUnBiMjR1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFFTTExVjUrTjJUWE5MMllBTDlLZlVNSlBmZytzZDFVaEdidDgyVDJHNmtSeHY5cwphNXFFTTU0L2h5MjVjT2xMc0doKzd1bnN0T0NwUnUxNlpGWUVtZy9rdlFyRmpYZ0o4K1l5eW9QSVEvUHM0MnhnCklPQTE0c0F5dUdWbTRxeFFSdGl0ZXVJcnlUbTJvZHozN3RjdmhvL3h3emQ4ay9SMDhhUkUzTld0ODlqR3orbkgKUG5HbERrVU9sKzFQRVZJei9tNkh2S2J6RDFWZnFqN2ltSlJPVVdLRWRJWEdWdk9sNmlpd1YweDlNZk1CWFNJSgpUNHB6c3RObE03WUlGcTBpWjMyMjdRYTdGKzdRZGpJNGpSbzQ0VVd4L1RiVTJVR2VqZ2JpY1VnL1ZOMkxCek1KCjR3Y25TVVFYQVNNQ1VVL0Rvb2NtZnZwQU1PRkhPRUZ1c0JhN0tIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Service:
Name: opentelemetry-operator-webhook-service
Namespace: application
Path: /mutate-v1-pod
Port: 443
Failure Policy: Ignore
Match Policy: Equivalent
Name: mpod.kb.io
Namespace Selector:
Object Selector:
Reinvocation Policy: Never
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
UPDATE
Resources:
pods
Scope: *
Side Effects: None
Timeout Seconds: 10
Events: <none>
There are a few events related to other things which are deployed, but nothing pertaining to this.
What about logs of opentelemetry-operator-controller-manager-...
?
Also Kubernetes API Server might have more details, so try to look at it's logs
@chris-minka did you get anywhere with this? I was seeing the same timeout error (I've just disabled webhooks for now).
@povilasv it seems that mopentelemetrycollector.kb.io
is coming from the chart, e.g. https://github.com/open-telemetry/opentelemetry-helm-charts/blob/56701aeb4c52a6fcdb459d3774dd91e6c67a723a/charts/opentelemetry-operator/templates/admission-webhooks/operator-webhook.yaml#L68
@milesarmstrong i ended up deploying the collector using its helm chart instead of using the operator. so, unfortunately i do not have a solution. if i were to revisit the issue, i would provide the logs @povilasv requested. also, since we are using a GKE Private Cluster i would review the firewall rules.
Thanks @chris-minka, we're also using GKE Private. I suspect it is the firewall rules, thanks for the pointer!
I posted this comment in the knative project, I think the issue is very similar: https://github.com/knative/serving/issues/13045#issuecomment-1359356226
We run our clusters on GKE. The webhook call is made from the apiserver. The webhook pod listens on 8443 whilst the service listens on 443. When making the webhook call the GKE apiserver tries to hit the webhook pod on 8443. Only ports 443 and 10250 are open between the apiserver and the GKE nodes.
I hit this issue also and digging into it, it is definitely a GKE firewall issue, but it is not a port 8443 issue.
When I set the allow to tcp:8443, the collector still would not build. When I opened all ports tcp:1-65535, collector did build.
So now I am trying to determine what actual port it needs and is getting blocked. I tried 8080 also but no luck. I'll report back when I figure it out (unless someone chimes in and knows what the port should be).
It's port 9443 for some reason (I brute forced found it by trial and error), even though the service describe says 8443.
adam@AdamPC:~/opentelemetry$ gcloud compute firewall-rules create gke-opentelemetry-webhook \ --action ALLOW \ --direction INGRESS \ --source-ranges 172.16.0.16/28 \ --rules tcp:9443 \ --target-tags gke-cluster-19216119-node Creating firewall... Creating firewall...done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED gke-opentelemetry-webhook default INGRESS 1000 tcp:9443 False adam@AdamPC:~/opentelemetry$ kubectl -n opentelemetry apply -f collector.yaml opentelemetrycollector.opentelemetry.io/simplest created
There is an arg for a webhook port on the operator Args: --webhook-port=9443
If that webhook port gets changed to 443, there shouldn't be a need for a special firewall rule, correct? Since the default GKE firewall would be sufficient.
I get the same error with version 0.36.0 of the opentelemetry-operator chart, and downgrading to 0.34.0 resolves it. I suspect the wrong webhook name was added in the newer version of the chart
I have deployed the operator (chart version 0.21.4) using the default chart values into a namespace called application. The pod starts and seems healthy. When I try deploy the collector there is an error:
collector.yaml:
The firewall allows traffic on 443. Also, a test pod does not timeout when calling the webhook URL directly: