i-chvets
commented
1 year ago On track/2.0 the following failures observed:
Model Controller Cloud/Region Version SLA Timestamp
test-charm-yjgo github-pr-77e6a-microk8s microk8s/localhost 2.9.44 unsupported 11:52:17Z
App Version Status Scale Charm Channel Rev Address Exposed Message
grafana-k8s 9.2.1 active 1 grafana-k8s latest/stable 81 10.152.183.115 no
kfp-api waiting 1 kfp-api 0 10.152.183.6 no installing agent
kfp-db mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s latest/stable 35 10.152.183.232 no ready
kfp-viz res:oci-image@3de6f3c blocked 0/1 kfp-viz 2.0/stable 476 10.152.183.34 no 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes ...
minio res:oci-image@1755999 blocked 0/1 minio ckf-1.7/stable 186 10.152.183.121 no 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes ...
mysql-k8s 8.0.32-0ubuntu0.22.04.2 waiting 1 mysql-k8s 8.0/edge 88 10.152.183.231 no waiting for units to settle down
prometheus-k8s 2.33.5 active 1 prometheus-k8s latest/stable 103 10.152.183.141 no
prometheus-scrape-config-k8s n/a active 1 prometheus-scrape-config-k8s latest/stable 39 10.152.183.123 no
Unit Workload Agent Address Ports Message
grafana-k8s/0\* active idle 10.1.76.158
kfp-api/0\* blocked idle 10.1.76.137 Please add required database relation: eg. relational-db
kfp-db/0\* active idle 10.1.76.145 3306/TCP ready
kfp-viz/0 unknown lost 10.1.76.146 8888/TCP agent lost, see 'juju show-status-log kfp-viz/0'
minio/0 unknown lost 10.1.76.149 9000/TCP,9001/TCP agent lost, see 'juju show-status-log minio/0'
mysql-k8s/0\* error idle 10.1.76.160 hook failed: "database-relation-changed"
prometheus-k8s/0\* active idle 10.1.76.157
prometheus-scrape-config-k8s/0\* active idle 10.1.76.153
DnPlas
ca-scribner
kimwnasptd
orfeas-k
phoevos
barrettj12
syncronize-issues-to-jira[bot]
It seems like there is an intermittent issue when deploying charms from Charmhub in
GH runners, which fail to deploy with the status message:unknown container reason "ImagePullBackOff": Back-off pulling image "registry.jujucharms.com/charm/4jn95wlrsfo1po9wfu....This error message suggests there is an error when pulling the image from the
registry.jujucharms.com/container registry, but this information has to be confirmed because these are sometimes linked to low storage.Important to note, this error goes away after re-running the CI, as mentioned before, it seems like an intermittent issue, but has happened consistently across other PRs at least once.
Environment
This has happened in GH runners during integration test execution, but also on machines with more resources and better network settings. The error is not related to one charm in particular, it has happened to different ones (minio, kubeflow-volumes, etc.)
Possible causes
It can be either the registry is not working correctly OR we could be running out of storage in the GH runners.
Examples of CIs where this behaviour has been observed
1) https://github.com/canonical/kfp-operators/actions/runs/5716319711/job/15514631356#step:8:38 2) https://github.com/canonical/kfp-operators/actions/runs/5726957265/job/15518414482#step:8:28 3) https://github.com/canonical/kfp-operators/actions/runs/5730163441/job/15533052295#step:5:77