Compass Integration job sometimes fails during Kyma installation

Description

Integration jobs that install Kyma + Compass in VM (using this script) seem to be sometimes randomly failing during the installation.

The prow script always ends during the Triggering the installation step, with 2h timeout, after logging:

- install component dex

After checking logs from one VM instance where this problem occurred, it looked like the installation finished successfully in time (see logs below, pay attention to the time of finished installation, and the timeout logged by prow script). This can mean, that the problem is related to gcloud compute ssh command used in the prow script or the connection between prow cluster and VM instance.

Example logs from failed job

``` [0;32m ################################################################################################# # Execute Job Guard ################################################################################################# [0m This jobguard location is deprecated. Please call jobguard from new location github.com/kyma-project/test-infra/development/jobguard KYMA PROJECT DIR /home/prow/go/src/github.com/kyma-project COMMIT SHA: 55423de828761f860a28a881505e67581f3012f1 go: downloading k8s.io/test-infra v0.0.0-20200331085241-bf7dc9346358 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/vrischmann/envconfig v1.2.0 go: extracting github.com/vrischmann/envconfig v1.2.0 go: extracting github.com/pkg/errors v0.9.1 go: extracting k8s.io/test-infra v0.0.0-20200331085241-bf7dc9346358 go: downloading k8s.io/apimachinery v0.16.5-beta.1 go: downloading github.com/tektoncd/pipeline v0.10.1 go: downloading gopkg.in/robfig/cron.v2 v2.0.0-20150107220207-be2e0b0deed5 go: downloading golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d go: downloading k8s.io/api v0.16.4 go: downloading sigs.k8s.io/yaml v1.1.0 go: downloading cloud.google.com/go v0.47.0 go: downloading google.golang.org/api v0.10.0 go: extracting sigs.k8s.io/yaml v1.1.0 go: extracting gopkg.in/robfig/cron.v2 v2.0.0-20150107220207-be2e0b0deed5 go: extracting golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d go: downloading github.com/GoogleCloudPlatform/testgrid v0.0.7 go: downloading k8s.io/client-go v0.16.4 go: extracting github.com/GoogleCloudPlatform/testgrid v0.0.7 go: extracting k8s.io/apimachinery v0.16.5-beta.1 go: extracting cloud.google.com/go v0.47.0 go: extracting github.com/tektoncd/pipeline v0.10.1 go: downloading golang.org/x/time v0.0.0-20191024005414-555d28b269f0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/fsnotify/fsnotify v1.4.7 go: downloading github.com/shurcooL/githubv4 v0.0.0-20191102174205-af46314aec7b go: extracting golang.org/x/time v0.0.0-20191024005414-555d28b269f0 go: extracting github.com/shurcooL/githubv4 v0.0.0-20191102174205-af46314aec7b go: extracting github.com/fsnotify/fsnotify v1.4.7 go: extracting gopkg.in/inf.v0 v0.9.1 go: downloading golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e go: downloading github.com/gogo/protobuf v1.3.1 go: extracting k8s.io/api v0.16.4 go: downloading github.com/google/gofuzz v1.0.0 go: extracting golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e go: extracting k8s.io/client-go v0.16.4 go: extracting github.com/google/gofuzz v1.0.0 go: downloading github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc go: extracting github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc go: downloading cloud.google.com/go/storage v1.0.0 go: downloading k8s.io/klog v1.0.0 go: extracting github.com/gogo/protobuf v1.3.1 go: extracting k8s.io/klog v1.0.0 go: extracting cloud.google.com/go/storage v1.0.0 go: downloading github.com/googleapis/gax-go v2.0.2+incompatible go: downloading github.com/google/go-cmp v0.4.0 go: downloading github.com/google/go-github v17.0.0+incompatible go: downloading knative.dev/pkg v0.0.0-20191111150521-6d806b998379 go: downloading golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527 go: extracting github.com/googleapis/gax-go v2.0.2+incompatible go: downloading github.com/googleapis/gax-go/v2 v2.0.5 go: extracting github.com/google/go-github v17.0.0+incompatible go: extracting github.com/googleapis/gax-go/v2 v2.0.5 go: downloading google.golang.org/grpc v1.27.0 go: extracting github.com/google/go-cmp v0.4.0 go: downloading golang.org/x/net v0.0.0-20200301022130-244492dfa37a go: extracting golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527 go: extracting knative.dev/pkg v0.0.0-20191111150521-6d806b998379 go: extracting google.golang.org/grpc v1.27.0 go: downloading github.com/imdario/mergo v0.3.8 go: extracting golang.org/x/net v0.0.0-20200301022130-244492dfa37a go: extracting github.com/imdario/mergo v0.3.8 go: downloading github.com/sirupsen/logrus v1.4.2 go: downloading sigs.k8s.io/controller-runtime v0.5.0 go: downloading gopkg.in/yaml.v2 v2.2.8 go: downloading github.com/google/go-querystring v1.0.0 go: downloading github.com/prometheus/client_golang v1.5.0 go: extracting gopkg.in/yaml.v2 v2.2.8 go: extracting github.com/sirupsen/logrus v1.4.2 go: extracting github.com/google/go-querystring v1.0.0 go: downloading github.com/gomodule/redigo v1.7.0 go: extracting github.com/prometheus/client_golang v1.5.0 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: extracting github.com/gomodule/redigo v1.7.0 go: extracting sigs.k8s.io/controller-runtime v0.5.0 go: downloading github.com/golang/protobuf v1.3.4 go: extracting github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/cespare/xxhash v1.1.0 go: extracting github.com/cespare/xxhash v1.1.0 go: extracting github.com/spf13/pflag v1.0.5 go: downloading github.com/cespare/xxhash/v2 v2.1.1 go: downloading github.com/beorn7/perks v1.0.1 go: extracting github.com/golang/protobuf v1.3.4 go: extracting google.golang.org/api v0.10.0 go: extracting github.com/beorn7/perks v1.0.1 go: downloading github.com/google/btree v1.0.0 go: extracting github.com/cespare/xxhash/v2 v2.1.1 go: downloading golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 go: downloading github.com/prometheus/common v0.9.1 go: downloading k8s.io/utils v0.0.0-20191114184206-e782cd3c129f go: downloading github.com/json-iterator/go v1.1.9 go: extracting github.com/google/btree v1.0.0 go: downloading github.com/gophercloud/gophercloud v0.1.0 go: extracting github.com/prometheus/common v0.9.1 go: extracting github.com/json-iterator/go v1.1.9 go: extracting k8s.io/utils v0.0.0-20191114184206-e782cd3c129f go: downloading github.com/gorilla/sessions v1.2.0 go: downloading google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03 go: downloading k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a go: downloading vbom.ml/util v0.0.0-20180919145318-efcd4e0f9787 go: extracting github.com/gorilla/sessions v1.2.0 go: downloading github.com/Azure/go-autorest v13.0.0+incompatible go: extracting golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 go: extracting github.com/Azure/go-autorest v13.0.0+incompatible go: downloading github.com/Azure/go-autorest/autorest v0.9.6 go: extracting vbom.ml/util v0.0.0-20180919145318-efcd4e0f9787 go: downloading github.com/prometheus/procfs v0.0.10 go: extracting github.com/Azure/go-autorest/autorest v0.9.6 go: extracting github.com/prometheus/procfs v0.0.10 go: downloading github.com/prometheus/client_model v0.2.0 go: extracting github.com/prometheus/client_model v0.2.0 go: extracting k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a go: extracting github.com/gophercloud/gophercloud v0.1.0 go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b go: extracting github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b go: downloading github.com/shurcooL/graphql v0.0.0-20181231061246-d48a9a75455f go: extracting github.com/shurcooL/graphql v0.0.0-20181231061246-d48a9a75455f go: downloading go.opencensus.io v0.22.3 go: downloading github.com/Azure/go-autorest/autorest/adal v0.8.2 go: extracting github.com/Azure/go-autorest/autorest/adal v0.8.2 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1 go: extracting go.opencensus.io v0.22.3 go: extracting github.com/matttproud/golang_protobuf_extensions v1.0.1 go: downloading github.com/gorilla/securecookie v1.1.1 go: extracting github.com/gorilla/securecookie v1.1.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: extracting github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/Azure/go-autorest/autorest/date v0.2.0 go: extracting github.com/Azure/go-autorest/autorest/date v0.2.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: extracting github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 go: extracting github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 go: downloading github.com/davecgh/go-spew v1.1.1 go: extracting github.com/davecgh/go-spew v1.1.1 go: downloading github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a go: extracting github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/googleapis/gnostic v0.3.1 go: downloading github.com/Azure/go-autorest/logger v0.1.0 go: extracting github.com/ghodss/yaml v1.0.0 go: downloading github.com/evanphx/json-patch v4.5.0+incompatible go: extracting github.com/Azure/go-autorest/logger v0.1.0 go: extracting github.com/evanphx/json-patch v4.5.0+incompatible go: downloading github.com/hashicorp/golang-lru v0.5.4 go: extracting github.com/googleapis/gnostic v0.3.1 go: extracting github.com/hashicorp/golang-lru v0.5.4 go: downloading github.com/modern-go/reflect2 v1.0.1 go: extracting github.com/modern-go/reflect2 v1.0.1 go: downloading golang.org/x/text v0.3.2 go: extracting google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03 go: downloading github.com/Azure/go-autorest/tracing v0.5.0 go: extracting github.com/Azure/go-autorest/tracing v0.5.0 go: extracting golang.org/x/text v0.3.2 go: finding github.com/pkg/errors v0.9.1 go: finding github.com/vrischmann/envconfig v1.2.0 go: finding k8s.io/test-infra v0.0.0-20200331085241-bf7dc9346358 go: finding github.com/sirupsen/logrus v1.4.2 go: finding golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527 go: finding github.com/tektoncd/pipeline v0.10.1 go: finding github.com/google/go-cmp v0.4.0 go: finding github.com/ghodss/yaml v1.0.0 go: finding gopkg.in/yaml.v2 v2.2.8 go: finding k8s.io/api v0.16.4 go: finding github.com/gogo/protobuf v1.3.1 go: finding k8s.io/apimachinery v0.16.5-beta.1 go: finding gopkg.in/inf.v0 v0.9.1 go: finding github.com/google/gofuzz v1.0.0 go: finding k8s.io/klog v1.0.0 go: finding golang.org/x/net v0.0.0-20200301022130-244492dfa37a go: finding golang.org/x/text v0.3.2 go: finding knative.dev/pkg v0.0.0-20191111150521-6d806b998379 go: finding k8s.io/client-go v0.16.4 go: finding github.com/golang/protobuf v1.3.4 go: finding github.com/googleapis/gnostic v0.3.1 go: finding github.com/json-iterator/go v1.1.9 go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: finding github.com/modern-go/reflect2 v1.0.1 go: finding sigs.k8s.io/yaml v1.1.0 go: finding github.com/davecgh/go-spew v1.1.1 go: finding golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 go: finding golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d go: finding golang.org/x/time v0.0.0-20191024005414-555d28b269f0 go: finding k8s.io/utils v0.0.0-20191114184206-e782cd3c129f go: finding github.com/hashicorp/golang-lru v0.5.4 go: finding k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a go: finding github.com/evanphx/json-patch v4.5.0+incompatible go: finding github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a go: finding gopkg.in/robfig/cron.v2 v2.0.0-20150107220207-be2e0b0deed5 go: finding github.com/shurcooL/githubv4 v0.0.0-20191102174205-af46314aec7b go: finding github.com/shurcooL/graphql v0.0.0-20181231061246-d48a9a75455f go: finding github.com/gomodule/redigo v1.7.0 go: finding github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc go: finding github.com/peterbourgon/diskv v2.0.1+incompatible go: finding github.com/google/btree v1.0.0 go: finding github.com/prometheus/client_golang v1.5.0 go: finding github.com/beorn7/perks v1.0.1 go: finding github.com/cespare/xxhash/v2 v2.1.1 go: finding github.com/prometheus/client_model v0.2.0 go: finding github.com/prometheus/common v0.9.1 go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1 go: finding github.com/prometheus/procfs v0.0.10 go: finding golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e go: finding github.com/Azure/go-autorest/autorest v0.9.6 go: finding github.com/Azure/go-autorest/autorest/adal v0.8.2 go: finding github.com/Azure/go-autorest/autorest/date v0.2.0 go: finding github.com/Azure/go-autorest/tracing v0.5.0 go: finding github.com/dgrijalva/jwt-go v3.2.0+incompatible go: finding github.com/Azure/go-autorest/logger v0.1.0 go: finding cloud.google.com/go v0.47.0 go: finding github.com/gophercloud/gophercloud v0.1.0 go: finding github.com/imdario/mergo v0.3.8 go: finding github.com/spf13/pflag v1.0.5 go: finding github.com/fsnotify/fsnotify v1.4.7 go: finding cloud.google.com/go/storage v1.0.0 go: finding github.com/googleapis/gax-go/v2 v2.0.5 go: finding google.golang.org/grpc v1.27.0 go: finding google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03 go: finding google.golang.org/api v0.10.0 go: finding go.opencensus.io v0.22.3 go: finding github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 go: finding github.com/GoogleCloudPlatform/testgrid v0.0.7 go: finding vbom.ml/util v0.0.0-20180919145318-efcd4e0f9787 go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b go: finding github.com/google/go-github v17.0.0+incompatible go: finding github.com/google/go-querystring v1.0.0 go: finding github.com/gorilla/sessions v1.2.0 go: finding github.com/gorilla/securecookie v1.1.1 go: finding sigs.k8s.io/controller-runtime v0.5.0 2020/05/05 05:12:12 Starting Job Guard... 2020/05/05 05:12:12 Sleeping for 1m0s... 2020/05/05 05:13:12 Expected jobs: [pre-compass-components-connectivity-adapter pre-compass-components-connector pre-compass-components-director pre-compass-components-external-services-mock pre-compass-components-gateway pre-compass-components-healthchecker pre-compass-components-kyma-environment-broker pre-compass-components-metris pre-compass-components-pairing-adapter pre-compass-components-provisioner pre-compass-components-schema-migrator pre-compass-tests-connectivity-adapter pre-compass-tests-connector-tests pre-compass-tests-director pre-compass-tests-e2e-provisioning pre-compass-tests-provisioner-tests] 2020/05/05 05:13:12 Fetched statuses: 20/20 2020/05/05 05:13:12 Waiting for jobs to finish: [pre-compass-components-external-services-mock,pre-compass-tests-director] 2020/05/05 05:13:27 Fetched statuses: 20/20 2020/05/05 05:13:27 Waiting for jobs to finish: [pre-compass-components-external-services-mock,pre-compass-tests-director] 2020/05/05 05:13:42 Fetched statuses: 20/20 2020/05/05 05:13:42 Waiting for jobs to finish: [pre-compass-components-external-services-mock,pre-compass-tests-director] 2020/05/05 05:13:57 Fetched statuses: 20/20 2020/05/05 05:13:57 Waiting for jobs to finish: [pre-compass-components-external-services-mock,pre-compass-tests-director] 2020/05/05 05:14:12 Fetched statuses: 20/20 2020/05/05 05:14:12 Waiting for jobs to finish: [pre-compass-components-external-services-mock,pre-compass-tests-director] 2020/05/05 05:14:27 Fetched statuses: 20/20 2020/05/05 05:14:27 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:14:42 Fetched statuses: 20/20 2020/05/05 05:14:42 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:14:57 Fetched statuses: 20/20 2020/05/05 05:14:57 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:15:12 Fetched statuses: 20/20 2020/05/05 05:15:12 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:15:27 Fetched statuses: 20/20 2020/05/05 05:15:27 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:15:42 Fetched statuses: 20/20 2020/05/05 05:15:42 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:15:57 Fetched statuses: 20/20 2020/05/05 05:15:57 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:16:12 Fetched statuses: 20/20 2020/05/05 05:16:12 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:16:27 Fetched statuses: 20/20 2020/05/05 05:16:27 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:16:42 Fetched statuses: 20/20 2020/05/05 05:16:42 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:16:57 Fetched statuses: 20/20 2020/05/05 05:16:57 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:17:12 Fetched statuses: 20/20 2020/05/05 05:17:12 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:17:27 Fetched statuses: 20/20 2020/05/05 05:17:27 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:17:42 Fetched statuses: 20/20 2020/05/05 05:17:42 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:17:57 Fetched statuses: 20/20 2020/05/05 05:17:57 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:18:12 Fetched statuses: 20/20 2020/05/05 05:18:12 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:18:27 Fetched statuses: 20/20 2020/05/05 05:18:27 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:18:42 Fetched statuses: 20/20 2020/05/05 05:18:42 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:18:57 Fetched statuses: 20/20 2020/05/05 05:18:57 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:19:12 Fetched statuses: 20/20 2020/05/05 05:19:12 Waiting for jobs to finish: [pre-compass-tests-director] 2020/05/05 05:19:27 Fetched statuses: 20/20 2020/05/05 05:19:27 [SUCCESS] All jobs with name matching pattern '(pre-compass-components-.*)|(pre-compass-tests-.*)' finished. Authenticating Activated service account credentials for: [sa-vm-kyma-integration@sap-kyma-prow.iam.gserviceaccount.com] [0;32m ################################################################################################# # Provisioning vm using the latest default custom image ... ################################################################################################# [0m [0;32m ################################################################################################# # Attempting to create a new instance named compass-integration-test-35c3ace0 in zone europe-west4-a using image kyma-deps-image-v20200206 ################################################################################################# [0m WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance. Created [https://www.googleapis.com/compute/v1/projects/sap-kyma-prow-workloads/zones/europe-west4-a/instances/compass-integration-test-35c3ace0]. NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS compass-integration-test-35c3ace0 europe-west4-a n1-standard-4 10.164.0.60 34.90.108.131 RUNNING [0;32m ################################################################################################# # Created compass-integration-test-35c3ace0 in zone europe-west4-a ################################################################################################# [0m [0;32m ################################################################################################# # Copying Compass to the instance ################################################################################################# [0m WARNING: The public SSH key file for gcloud does not exist. WARNING: The private SSH key file for gcloud does not exist. WARNING: You do not have an SSH key for gcloud. WARNING: SSH keygen will be executed to generate a key. Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/google_compute_engine. Your public key has been saved in /root/.ssh/google_compute_engine.pub. The key fingerprint is: SHA256:OJ7DGmjMXLNyCfmYcCwDDZovdTQ/OMzJ0Np5AC4xqnY root@99340379-8e8e-11ea-9809-42b8f52df38b The key's randomart image is: +---[RSA 2048]----+ |+ ooo | |oB *o= | |* +oOoo | |o=.oo..o | |*.*Eo.o S | |.X.B * o | | X * = | | . o o . | | . | +----[SHA256]-----+ Warning: Permanently added 'compute.5953334267980106123' (ECDSA) to the list of known hosts. Permission denied (publickey). lost connection ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. Retrying in 15 seconds.. [0;32m ################################################################################################# # Download stable Kyma CLI ################################################################################################# [0m % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 22 64.4M 22 14.2M 0 0 61.7M 0 0:00:01 --:--:-- 0:00:01 61.7M 100 64.4M 100 64.4M 0 0 176M 0 --:--:-- --:--:-- --:--:-- 176M Pseudo-terminal will not be allocated because stdin is not a terminal. Pseudo-terminal will not be allocated because stdin is not a terminal. [0;32m ################################################################################################# # Triggering the installation ################################################################################################# [0m Pseudo-terminal will not be allocated because stdin is not a terminal. - Requirements verified Preparing Minikube using vm-driver 'none' - Minikube status verified - Minikube config initialized Create Minikube instance: Start Minikube Create Minikube instance: Wait for Minikube to be up and running Create Minikube instance: Running Create Minikube instance: Running Create Minikube instance: Create default cluster role Create Minikube instance: Wait for kube-dns to be up and running - Minikube up and running Adjusting Minikube cluster: Increase fs.inotify.max_user_instances Adjusting Minikube cluster: Enable metrics server - Adjustments finished - ConfigMap created Minikube cluster installed host: Running kubelet: Running apiserver: Running kubeconfig: Configured Happy Minikube-ing! :) - Cluster info read - Configurations validated Using the installation configuration from 'master' Installing Kyma in version 'master-7d00def9' - Installation source checked - Tiller deployed - Installation files loaded - Kyma Installer deployed - Minikube IP added - Helm configured - Kyma Installer is installing Kyma - Waiting for installation to start - install component cluster-essentials - install component testing - install component istio - install component istio-kyma-patch - install component dex {"component":"entrypoint","file":"prow/entrypoint/run.go:164","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","time":"2020-05-05T07:10:03Z"} {"component":"entrypoint","file":"prow/entrypoint/run.go:245","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 10m0s grace period","time":"2020-05-05T07:20:03Z"} ```

Example logs from installer in VM instance

``` t_gorgol_sap_com@compass-integration-test-35c3ace0:~$ sudo kubectl logs -n kyma-installer kyma-installer-f4fdffc68-7tpkm 2020/05/05 05:22:20 Starting operator... 2020/05/05 05:22:24 Processing Kyma components 2020/05/05 05:22:24 --------------------------- 2020/05/05 05:22:24 install component cluster-essentials 2020/05/05 05:22:24 --------------------------- 2020/05/05 05:22:24 Get Kyma Sources 2020/05/05 05:22:24 Kyma sources available locally. 2020/05/05 05:22:24 Get Kyma Sources...DONE 2020/05/05 05:22:32 Name: cluster-essentials 2020/05/05 05:22:32 Namespace: kyma-system 2020/05/05 05:22:32 Version: 1 2020/05/05 05:22:32 Status: DEPLOYED 2020/05/05 05:22:32 Description: Install complete 2020/05/05 05:22:32 install component cluster-essentials...DONE! 2020/05/05 05:22:32 --------------------------- 2020/05/05 05:22:32 install component testing 2020/05/05 05:22:32 --------------------------- 2020/05/05 05:22:32 Get Kyma Sources 2020/05/05 05:22:32 Kyma sources available locally. 2020/05/05 05:22:32 Get Kyma Sources...DONE 2020/05/05 05:22:36 Name: testing 2020/05/05 05:22:36 Namespace: kyma-system 2020/05/05 05:22:36 Version: 1 2020/05/05 05:22:36 Status: DEPLOYED 2020/05/05 05:22:36 Description: Install complete 2020/05/05 05:22:36 install component testing...DONE! 2020/05/05 05:22:36 --------------------------- 2020/05/05 05:22:36 install component istio 2020/05/05 05:22:36 --------------------------- 2020/05/05 05:22:36 Get Kyma Sources 2020/05/05 05:22:36 Kyma sources available locally. 2020/05/05 05:22:36 Get Kyma Sources...DONE 2020/05/05 05:23:49 Name: istio 2020/05/05 05:23:49 Namespace: istio-system 2020/05/05 05:23:49 Version: 1 2020/05/05 05:23:49 Status: DEPLOYED 2020/05/05 05:23:49 Description: Install complete 2020/05/05 05:23:49 install component istio...DONE! 2020/05/05 05:23:49 --------------------------- 2020/05/05 05:23:49 install component xip-patch 2020/05/05 05:23:49 --------------------------- 2020/05/05 05:23:49 Get Kyma Sources 2020/05/05 05:23:49 Kyma sources available locally. 2020/05/05 05:23:49 Get Kyma Sources...DONE 2020/05/05 05:23:56 Name: xip-patch 2020/05/05 05:23:56 Namespace: kyma-installer 2020/05/05 05:23:56 Version: 1 2020/05/05 05:23:56 Status: DEPLOYED 2020/05/05 05:23:56 Description: Install complete 2020/05/05 05:23:56 install component xip-patch...DONE! 2020/05/05 05:23:56 --------------------------- 2020/05/05 05:23:56 install component istio-kyma-patch 2020/05/05 05:23:56 --------------------------- 2020/05/05 05:23:56 Get Kyma Sources 2020/05/05 05:23:56 Kyma sources available locally. 2020/05/05 05:23:56 Get Kyma Sources...DONE 2020/05/05 05:24:04 Name: istio-kyma-patch 2020/05/05 05:24:04 Namespace: istio-system 2020/05/05 05:24:04 Version: 1 2020/05/05 05:24:04 Status: DEPLOYED 2020/05/05 05:24:04 Description: Install complete 2020/05/05 05:24:04 install component istio-kyma-patch...DONE! 2020/05/05 05:24:04 --------------------------- 2020/05/05 05:24:04 install component dex 2020/05/05 05:24:04 --------------------------- 2020/05/05 05:24:04 Get Kyma Sources 2020/05/05 05:24:04 Kyma sources available locally. 2020/05/05 05:24:04 Get Kyma Sources...DONE 2020/05/05 05:24:39 Name: dex 2020/05/05 05:24:39 Namespace: kyma-system 2020/05/05 05:24:39 Version: 1 2020/05/05 05:24:39 Status: DEPLOYED 2020/05/05 05:24:39 Description: Install complete 2020/05/05 05:24:39 install component dex...DONE! 2020/05/05 05:24:39 --------------------------- 2020/05/05 05:24:39 install component ory 2020/05/05 05:24:39 --------------------------- 2020/05/05 05:24:39 Get Kyma Sources 2020/05/05 05:24:39 Kyma sources available locally. 2020/05/05 05:24:39 Get Kyma Sources...DONE 2020/05/05 05:40:18 Name: ory 2020/05/05 05:40:18 Namespace: kyma-system 2020/05/05 05:40:18 Version: 1 2020/05/05 05:40:18 Status: DEPLOYED 2020/05/05 05:40:18 Description: Install complete 2020/05/05 05:40:18 install component ory...DONE! 2020/05/05 05:40:18 install Kyma components ...DONE! ```

Example installation CR in VM instance

``` t_gorgol_sap_com@compass-integration-test-35c3ace0:~$ sudo kubectl describe installations Name: kyma-installation Namespace: default Labels: kyma-project.io/installation= Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"installer.kyma-project.io/v1alpha1","kind":"Installation","metadata":{"annotations":{},"finalizers":["finalizer.installer.k... API Version: installer.kyma-project.io/v1alpha1 Kind: Installation Metadata: Creation Timestamp: 2020-05-05T05:22:16Z Finalizers: finalizer.installer.kyma-project.io Generation: 13 Resource Version: 3400 Self Link: /apis/installer.kyma-project.io/v1alpha1/namespaces/default/installations/kyma-installation UID: 12d115f7-fd15-43e1-98d2-0f5c1029cb58 Spec: Components: Name: cluster-essentials Namespace: kyma-system Release: Name: testing Namespace: kyma-system Release: Name: istio Namespace: istio-system Release: Name: xip-patch Namespace: kyma-installer Release: Name: istio-kyma-patch Namespace: istio-system Release: Name: dex Namespace: kyma-system Release: Name: ory Namespace: kyma-system Release: URL: __URL__ Version: __VERSION__ Status: Conditions: Last Probe Time: 2020-05-05T05:40:18Z Last Transition Time: 2020-05-05T05:40:18Z Status: True Type: Installed Last Probe Time: 2020-05-05T05:40:18Z Last Transition Time: 2020-05-05T05:40:18Z Status: False Type: Installing Last Probe Time: 2020-05-05T05:40:18Z Last Transition Time: 2020-05-05T05:40:18Z Status: False Type: InProgress Last Probe Time: 2020-05-05T05:40:18Z Last Transition Time: 2020-05-05T05:22:24Z Status: False Type: Error Last Probe Time: 2020-05-05T05:40:18Z Last Transition Time: 2020-05-05T05:40:18Z Status: False Type: Uninstalled Description: Kyma installed Error Log: State: Installed URL: __URL__ Version: __VERSION__ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Synced 3m52s (x375 over 3h26m) KymaOperator Installation synced successfully ```

Expected result

Prow job should continue logging installation progress and see that the installation was successfull.

Actual result

Prow job stops logging installation progress, after 2h there is a timeout.

Steps to reproduce

Seems to be happening randomly, on jobs installing Kyma + Compass in VM.

kyma-incubator / compass

Compass Integration job sometimes fails during Kyma installation #1290