search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
332 stars 162 forks source link

Flake - random tests are failing due to timeout on write #1996

Closed zimnx closed 2 months ago

zimnx commented 3 months ago

This was on -clusterip job which has slower persistent network attached ssds. We might want to reevaluate whether we still want to use them.

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1991/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1806721597633990656

Snippet of what failed.

   [FAILED] Unexpected error:
      <*fmt.wrapError | 0xc0006967a0>: 
      can't insert data: Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.
      {
          msg: "can't insert data: Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.",
          err: <*gocql.RequestErrWriteTimeout | 0xc0004f4380>{
              errorFrame: {
                  frameHeader: {version: 132, flags: 0, stream: 576, op: 0, length: 104, warnings: nil},
                  code: 4352,
                  message: "Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.",
              },
              Consistency: 5,
              Received: 1,
              BlockFor: 2,
              WriteType: "SIMPLE",
          },
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/verify.go:312 @ 06/28/24 16:30:32.464
zimnx commented 3 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1991/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1806749115925663744

tnozicka commented 3 months ago

We don' have a choice on some platforms - preferably we'd adjust the timeout or concurrency based o what exactly goes so slow. /priority important-longterm /triage accepted

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1871/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1808031823570145280#1:test-build-log.txt%3A1205

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810558454176157696#1:test-build-log.txt%3A962

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810586982485594112#1:test-build-log.txt%3A750

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810605342397042688#1:test-build-log.txt%3A1258