search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
333 stars 164 forks source link

Flake - should setup and maintain up to date TLS certificates #2096

Open zimnx opened 1 month ago

zimnx commented 1 month ago

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2066/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1828179211538403328

Snippet of what failed.

 ScyllaCluster [It] should setup and maintain up to date TLS certificates
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:42
  Timeline >>
  STEP: Creating a new namespace @ 08/26/24 21:30:43.436
  Aug 26 21:30:43.466: INFO: Created namespace "e2e-test-scyllacluster-2ms98-0-85njt".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:43.522
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.178
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.492
  STEP: Creating an initial ScyllaCluster with a single node @ 08/26/24 21:30:44.502
  STEP: Waiting for the ScyllaCluster to roll out (RV=5639) @ 08/26/24 21:30:44.533
  Aug 26 21:35:13.491: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11632) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:13.491
  Aug 26 21:35:13.648: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:13.648: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:13.757
  Aug 26 21:35:14.163: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Inserting data @ 08/26/24 21:35:14.259
  Aug 26 21:35:14.259: INFO: Creating CQL session (hosts="10.17.250.51")
  STEP: Inserting data @ 08/26/24 21:35:14.263
  Aug 26 21:35:14.263: INFO: Creating keyspace "6pztcv98" with RF "'replication_factor': 1"
  Aug 26 21:35:14.271: INFO: Creating table "6pztcv98"."test"
  Aug 26 21:35:14.281: INFO: Inserting data into table "6pztcv98"."test"
  Aug 26 21:35:14.296: INFO: Awaiting schema agreement
  Aug 26 21:35:14.297: INFO: Schema agreement reached
  STEP: Verifying the data @ 08/26/24 21:35:14.297
  Aug 26 21:35:14.297: INFO: Reading data from table "6pztcv98"."test"
  STEP: Scaling the ScyllaCluster to 1 replicas setting domains to [] @ 08/26/24 21:35:14.3
  STEP: Waiting for the ScyllaCluster to roll out (RV=11670) @ 08/26/24 21:35:14.496
  Aug 26 21:35:14.643: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11670) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:14.648
  Aug 26 21:35:14.905: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:14.905: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:14.981
  Aug 26 21:35:15.253: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:35:15.326
  STEP: Verifying certificates @ 08/26/24 21:35:15.384
  Aug 26 21:35:15.456: INFO: Starting to probe node "10.17.250.51" for correct certs
  Aug 26 21:35:15.530: INFO: Node "10.17.250.51" has correct certs
  STEP: Scaling the ScyllaCluster to 1 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:35:15.53
  STEP: Waiting for the ScyllaCluster to roll out (RV=11738) @ 08/26/24 21:35:15.601
  Aug 26 21:35:16.039: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11775) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:16.039
  Aug 26 21:35:16.296: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:16.296: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:16.458
  Aug 26 21:35:16.644: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:35:16.702
  STEP: Verifying certificates @ 08/26/24 21:35:16.856
  Aug 26 21:35:16.927: INFO: Starting to probe node "10.17.250.51" for correct certs
  Aug 26 21:36:11.572: INFO: Node "10.17.250.51" has correct certs
  STEP: Scaling the ScyllaCluster to 2 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:36:11.572
  STEP: Waiting for the ScyllaCluster to roll out (RV=13780) @ 08/26/24 21:36:11.604
  Aug 26 21:37:44.388: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=16384) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:37:44.389
  Aug 26 21:37:44.431: INFO: Found 2 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:37:44.431: INFO: Found 2 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:37:44.496
  Aug 26 21:37:44.576: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:37:44.608
  STEP: Verifying certificates @ 08/26/24 21:37:44.654
  [FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
  STEP: Collecting events from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.68
  STEP: Found 85 events. @ 08/26/24 21:37:44.694
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterCreated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:49 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:50 +0000 UTC - event for basic-nztd8-auth-token: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-auth-token created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:41 +0000 UTC - event for basic-nztd8-local-user-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-user-admin created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {scylladbdatacenter-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {controllermanager } NoPods: No matching pods found
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-client: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-client created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-managed-config: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-managed-config created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-0 Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-0"
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-80a399a7-965f-4ccb-b26d-9a14946e88ea
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:26 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:02 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:16 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:32 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:8080: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.50:42081/readyz": dial tcp 10.125.193.50:42081: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:10001: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:52 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterUpdated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:11 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-1 Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-ebb938df-3693-4f56-adb3-fc8ef5f6cb72
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-1"
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:45 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:47 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:8080: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:10001: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.92:42081/readyz": dial tcp 10.125.193.92:42081: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetUpdated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for basic-nztd8: {controllermanager } CalculateExpectedPodCountFailed: Failed to calculate the number of expected pods: jobs.batch does not implement the scale subresource
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } JobCreated: Job e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.694
  STEP: Destroying namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:45.437
  STEP: Waiting for namespace "e2e-test-scyllacluster-2ms98-0-85njt" to be removed. @ 08/26/24 21:37:45.449
  << Timeline
  [FAILED] Expected
      <[]string | len:5, cap:5>: ["10.17.250.51", "10.17.255.223", "10.125.193.50", "10.125.193.92", ""]
  to have length 4
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1.1({{0xc000cbbdc0?, 0x1?, 0x0?}, 0xc000bd6708?}, 0xc000a2dcf0, 0xc0007c7500, {0x35a2038, 0xc0005c5810}, {0xc0007f3040, 0x1, ...})
        github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 +0x21c9
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1()
        github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:259 +0x930

Looks like a cleanup Pod - triggered after considering ScyllaCluster as rolled out (it's an expected edge case) - got picked up by the test logic and IP of it (empty one) failed the assertion.

scylla-operator-bot[bot] commented 2 weeks ago

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

/lifecycle stale

zimnx commented 1 week ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2137/pull-scylla-operator-master-e2e-gke-parallel/1841481213232025600

/remove-lifecycle stale

rzetelskik commented 3 days ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2142/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1843940515221868544#1:test-build-log.txt%3A1226