search

kyma-project / test-infra

Test infrastructure for the Kyma project.
https://status.build.kyma-project.io/
Apache License 2.0
38 stars 181 forks source link

configmaps "kube-dns" not found (issue in integration jobs) #2601

Closed adamwalach closed 4 years ago

adamwalach commented 4 years ago

Description

After upgrade to k8s 1.16 we found that some of our integration jobs fail with following message:

Created [https://container.googleapis.com/v1beta1/projects/sap-kyma-prow-workloads/zones/europe-west4-a/clusters/gke-upgrade-pr-9013-eunavrped2].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west4-a/gke-upgrade-pr-9013-eunavrped2?project=sap-kyma-prow-workloads
kubeconfig entry generated for gke-upgrade-pr-9013-eunavrped2.
NAME                            LOCATION        MASTER_VERSION  MASTER_IP      MACHINE_TYPE   NODE_VERSION   NUM_NODES  STATUS
gke-upgrade-pr-9013-eunavrped2  europe-west4-a  1.16.11-gke.5   34.91.235.130  n1-standard-4  1.16.11-gke.5  3          RUNNING
Error from server (NotFound): configmaps "kube-dns" not found

It happened few times already: Screenshot 2020-07-16 at 14 06 04

Example: https://storage.googleapis.com/kyma-prow-logs/pr-logs/pull/kyma-project_kyma/9013/pre-master-kyma-gke-upgrade/1283504073558986752/build-log.txt

Expected result

We should wait till the config map is created before applying our patch

Steps to reproduce

Random error, difficult to reproduce

Troubleshooting

adamwalach commented 4 years ago

After we merged #2616 I see no entries for "kube-dns" not found, but I was able to find Waiting for kube-dns to be available:

Screenshot 2020-07-24 at 13 38 02

I can't find any entries for kube-dns configmap not available, so I assume that 45s timeout is enough.

adamwalach commented 4 years ago

I found some today: https://storage.googleapis.com/kyma-prow-logs/pr-logs/pull/kyma-project_kyma/9108/pre-master-kyma-gke-integration/1286639744196808704/build-log.txt

---> Patching kube-dns
Error from server (NotFound): configmaps "kube-dns" not found
Waiting for kube-dns to be available. Try 1 of 3
Error from server (NotFound): configmaps "kube-dns" not found
Waiting for kube-dns to be available. Try 2 of 3
Error from server (NotFound): configmaps "kube-dns" not found
Waiting for kube-dns to be available. Try 3 of 3
configmap/kube-dns patched

Please note that it works properly, but the error message is still there. Maybe it would be good to suppress them, so we can easily search for issues that are really causing problems.