search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
337 stars 175 forks source link

Flake - ScyllaCluster [It] should re-bootstrap from old PVCs #2127

Open zimnx opened 1 month ago

zimnx commented 1 month ago

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/logs/ci-scylla-operator-enterprise-latest-e2e-gke-arm64-parallel/1838096006806245376

Snippet of what failed.

 ScyllaCluster [It] should re-bootstrap from old PVCs
github.com/scylladb/scylla-operator@v1.14.0-alpha.0.0.20240816132056-b3da0a4b8e93/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:25
  Timeline >>
  STEP: Creating a new namespace @ 09/23/24 06:12:16.787
  Sep 23 06:12:16.807: INFO: Created namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:16.887
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:16.954
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:17.007
  STEP: Creating a ScyllaCluster @ 09/23/24 06:12:17.073
  STEP: Waiting for the ScyllaCluster to roll out (RV=5815) @ 09/23/24 06:12:17.267
  [FAILED] in [It] - github.com/scylladb/scylla-operator@v1.14.0-alpha.0.0.20240816132056-b3da0a4b8e93/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 @ 09/23/24 06:37:47.269
  STEP: Collecting events from namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:47.27
  STEP: Found 124 events. @ 09/23/24 06:37:47.288
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-auth-token created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-member created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-member created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:28 +0000 UTC - event for basic-f4qbj-local-client-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-client-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:28 +0000 UTC - event for basic-f4qbj-local-client-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-client-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:12 +0000 UTC - event for basic-f4qbj-local-user-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-user-admin created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:52 +0000 UTC - event for basic-f4qbj-local-serving-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:52 +0000 UTC - event for basic-f4qbj-local-serving-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj: {controllermanager } NoPods: No matching pods found
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-client created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-managed-config: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-managed-config created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-local-cql-connection-configs-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-cql-connection-configs-admin created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-local-serving-certs: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-certs created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-0 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-0 Pod basic-f4qbj-us-east-1-us-east-1a-0 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-4520cd7b-0b87-4440-a7c1-a277b05b3c8a
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-0"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:27 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:27 +0000 UTC - event for nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:28 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:28 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:29 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:45 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.40:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.40:42081/readyz": dial tcp 10.70.1.40:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.40:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:08 +0000 UTC - event for basic-f4qbj-local-serving-certs: {scyllacluster-controller } SecretUpdated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-certs updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-1 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-1 Pod basic-f4qbj-us-east-1-us-east-1a-1 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-1"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-fa60535d-cd68-430f-9ec5-7cf99c2f4ad6
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:38 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:38 +0000 UTC - event for nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:41 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:59 +0000 UTC - event for nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.51:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.51:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.51:42081/readyz": dial tcp 10.70.1.51:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:08 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Startup probe failed: Get "http://10.70.1.51:8080/healthz": dial tcp 10.70.1.51:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-2 Pod basic-f4qbj-us-east-1-us-east-1a-2 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-2 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-2"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-4ee228a5-62ef-4c88-bc26-0bbfd1338c46
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:40 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:54 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:54 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.59:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.59:42081/readyz": dial tcp 10.70.1.59:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.59:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Startup probe failed: Get "http://10.70.1.59:8080/healthz": dial tcp 10.70.1.59:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:58 +0000 UTC - event for nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } JobCreated: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } JobDeleted: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0 deleted
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {job-controller } Completed: Job completed
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } JobDeleted: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-1 deleted
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } DeleteJobFailed: Failed to delete Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-1: jobs.batch "cleanup-basic-f4qbj-us-east-1-us-east-1a-1" not found
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:47.288
  STEP: Destroying namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:48.183
  STEP: Waiting for namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h" to be removed. @ 09/23/24 06:37:48.194
  << Timeline
  [FAILED] Unexpected error:
      <*fmt.wrapError | 0x4000a200e0>: 
      waiting has been interupted ([false]): timed out waiting for the condition
      {
          msg: "waiting has been interupted ([false]): timed out waiting for the condition",
          err: <wait.errInterrupted>{
              cause: <*errors.errorString | 0x4000334890>{
                  s: "timed out waiting for the condition",
              },
          },
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator@v1.14.0-alpha.0.0.20240816132056-b3da0a4b8e93/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 @ 09/23/24 06:37:47.269
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func10.1()
        github.com/scylladb/scylla-operator@v1.14.0-alpha.0.0.20240816132056-b3da0a4b8e93/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 +0x344
scylla-operator-bot[bot] commented 1 week ago

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

/lifecycle stale