Closed alekseytivonchik closed 6 months ago
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5340.
This message was autogenerated
Hitting the same issue
I am hitting the same in the customer's environment.
Kubernetes: Charmed Kubeflow (1.29 charms, 1.28 API) Juju: 3.1.7-genericlinux-amd64 Kubeflow bundle: https://pastebin.canonical.com/p/d82h7ygYQr/
Here is my debug-log for webhook: https://pastebin.canonical.com/p/RSpgx2yD6b/
Pod: $ kubectl get po admission-webhook-0 -n kubeflow NAME READY STATUS RESTARTS AGE admission-webhook-0 2/2 Running 0 21h
Can you confirm if you see in the juju debug-log the WARN
log: Cannot upload certificates to container, deferring
I do not see it in my juju debug-log also for all juju applications.
The SSL cert in my environment is provided and terminated via F5.
Can you also provide:
kubectl logs admission-webhook-0 -n kubeflow -c admission-webhook
kubectl logs jupyter-controller-0 -n kubeflow -c jupyter-controller
Admission-webhook logs: https://pastebin.canonical.com/p/JyHD6Mtctp/ Jupyter-controller logs: https://pastebin.canonical.com/p/FBMRGbptyz/ Services in kubeflow ns:
$ kubectl get po -n kubeflow
NAME READY STATUS RESTARTS AGE
admission-webhook-0 2/2 Running 0 4h
argo-controller-0 2/2 Running 0 25h
dex-auth-0 2/2 Running 0 25h
envoy-5fbb9ccbf9-7xhhc 1/1 Running 0 25h
envoy-operator-0 1/1 Running 0 25h
grafana-agent-k8s-0 2/2 Running 0 25h
istio-ingressgateway-0 1/1 Running 0 25h
istio-ingressgateway-workload-696584c7c5-2tv9s 1/1 Running 0 25h
istio-pilot-0 1/1 Running 0 25h
istiod-7f696dd599-ptkrn 1/1 Running 0 25h
jupyter-controller-0 2/2 Running 0 25h
jupyter-ui-0 2/2 Running 0 25h
katib-controller-5cc6d58bfd-6k7tw 1/1 Running 0 25h
katib-controller-operator-0 1/1 Running 0 25h
katib-db-0 2/2 Running 0 25h
katib-db-manager-0 2/2 Running 0 25h
katib-ui-0 2/2 Running 3 (23h ago) 25h
kfp-api-0 2/2 Running 2 (24h ago) 25h
kfp-db-0 2/2 Running 0 25h
kfp-metadata-writer-0 2/2 Running 0 25h
kfp-persistence-0 2/2 Running 0 25h
kfp-profile-controller-0 2/2 Running 0 25h
kfp-schedwf-0 2/2 Running 0 25h
kfp-ui-0 2/2 Running 1 (24h ago) 25h
kfp-viewer-0 2/2 Running 0 25h
kfp-viz-0 2/2 Running 2 (2d21h ago) 2d22h
knative-eventing-0 1/1 Running 0 25h
knative-operator-0 3/3 Running 0 25h
knative-serving-0 1/1 Running 1 (24h ago) 25h
kserve-controller-0 3/3 Running 1 (2d21h ago) 2d22h
kubeflow-dashboard-0 2/2 Running 0 25h
kubeflow-profiles-0 3/3 Running 7 (26h ago) 2d22h
kubeflow-roles-0 1/1 Running 0 25h
kubeflow-volumes-84f58746bf-rxr68 1/1 Running 0 25h
kubeflow-volumes-operator-0 1/1 Running 0 25h
metacontroller-operator-0 1/1 Running 0 25h
metacontroller-operator-charm-0 1/1 Running 0 25h
minio-0 1/1 Running 0 2d22h
minio-operator-0 1/1 Running 0 25h
mlflow-mysql-0 2/2 Running 0 2d22h
mlflow-server-0 3/3 Running 0 25h
mlmd-0 1/1 Running 0 2d22h
mlmd-operator-0 1/1 Running 0 25h
modeloperator-56c9578cfb-nbrq5 1/1 Running 0 25h
oidc-gatekeeper-0 2/2 Running 0 2d22h
pvcviewer-operator-0 3/3 Running 0 25h
resource-dispatcher-0 2/2 Running 0 25h
seldon-controller-manager-0 2/2 Running 4 (2d21h ago) 2d22h
tensorboard-controller-0 2/2 Running 0 25h
tensorboards-web-app-0 2/2 Running 0 25h
training-operator-0 2/2 Running 0 25h
@Barteus are you able to get the juju debug log for admission webhook from the very beginning?
I need to see which hook failed to execute
the debug logs you've attached are not from when the charm was deployed, you should see there juju.worker.uniter unit "admission-webhook/0" started
The status of the charm is:
$ juju status admission-webhook
Model Controller Cloud/Region Version SLA Timestamp
kubeflow foundations-maas ck8s/default 3.1.7 unsupported 14:12:33Z
SAAS Status Store URL
grafana active foundations-maas admin/cos.grafana
loki active foundations-maas admin/cos.loki
prometheus active foundations-maas admin/cos.prometheus
App Version Status Scale Charm Channel Rev Address Exposed Message
admission-webhook waiting 1 admission-webhook 1.8/stable 275 172.30.181.94 no waiting for units to settle down
Unit Workload Agent Address Ports Message
admission-webhook/0* maintenance idle 10.128.236.170 Workload failed health check
To me, it looks like all hooks went ok, and the issue occurred later.
Workaround to unstuck the admission-webhook:
juju ssh admission-webhook/0
bash
export CONTAINER_NAME=admission-webhook
export PEBBLE_SOCKET=/charm/containers/$CONTAINER_NAME/pebble.socket
alias pebble=/charm/bin/pebble
# pebble plan
pebble services
pebble restart admission-webhook
After few minutes all notebooks are running!
Big thanks to @NohaIhab and @ca-scribner for help!
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5392.
This message was autogenerated
the fix #125 is now merged and promoted to 1.8/stable
You can now refresh your charms, for reference the new revision is 301
.
cc: @natalytvinova @Barteus @alekseytivonchik
Bug Description
when creating a Notebook, I get the error:
statefulset/tst-note-t: create Pod tst-note-t-0 in StatefulSet tst-note-t failed error: Internal error occurred: failed calling webhook "admission-webhook.kubeflow.org": failed to call webhook: Post "https://admission-webhook.kubeflow.svc:4443/apply-poddefault?timeout=10s": dial tcp 10.233.59.161:4443: connect: connection refused
To Reproduce
snap install juju juju bootstrap my-k8s juju add-model kubeflow juju deploy kubeflow --trust
Environment
K8s: v1.28.6 OS: Ubuntu 22.04 Juju: 3.3.1-genericlinux-amd64
admission-webhook: 1.8/stable argo-controller: 3.3.10/stable dex-auth: 2.36/stable envoy: 2.0/stable istio-ingressgateway: 1.17/stable istio-pilot: 1.17/stable jupyter-controller: 1.8/stable jupyter-ui: 1.8/stable katib-controller: 0.16/stable katib-db: 8.0/stable katib-db-manager: 0.16/stable katib-ui: 0.16/stable kfp-api: 2.0/stable kfp-db: 8.0/stable kfp-metadata-writer: 2.0/stable kfp-persistence: 2.0/stable kfp-profile-controller: 2.0/stable kfp-schedwf: 2.0/stable kfp-ui: 2.0/stable kfp-viewer: 2.0/stable kfp-viz: 2.0/stable knative-eventing: 1.10/stable knative-operator: 1.10/stable knative-serving: 1.10/stable kserve-controller: 0.11/stable kubeflow-dashboard: 1.8/stable kubeflow-profiles: 1.8/stable kubeflow-roles: 1.8/stable kubeflow-volumes: 1.8/stable metacontroller-operator: 3.0/stable minio: ckf-1.8/stable mlmd: 1.14/stable oidc-gatekeeper: ckf-1.8/stable pvcviewer-operator: 1.8/stable seldon-controller-manager: 1.17/stable tensorboard-controller: 1.8/stable tensorboards-web-app: 1.8/stable training-operator: 1.7/stable
Relevant Log Output
Additional Context
No response