search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
336 stars 174 forks source link

[Flake] Scylla Manager integration should register cluster, sync backup tasks and support manual restore procedure [It] using default ScyllaDB version [RequiresObjectStorage] #2061

Open tnozicka opened 2 months ago

tnozicka commented 2 months ago

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2055/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1820487766287323136#1:test-build-log.txt%3A1002

Snippet of what failed.

   [FAILED] %!(EXTRA string=Error: create restore target, units and views: init target: look for schema file: get schema file from gcs:so-e490a80d-2b84-4d9d-a784-d0d9d7c27ee3/backup/schema: many schema files with sm_20240805161410UTC snapshot tag: [cluster/6f019377-acd2-4f23-9aad-d6cb21a05008/task_a7b7283e-c7eb-4830-a975-4fd5d16f19af_tag_sm_20240805161410UTC_schema_with_internals.json.gz cluster/82d8dd25-42fc-4c83-87a9-1e76a0e6a15e/task_d24f0a86-f287-4e04-87f8-46220bddda35_tag_sm_20240805161410UTC_schema_with_internals.json.gz]
  Trace ID: EmDCY15VQlG_ct5tWy6W_A (grep in scylla-manager logs)
  )
  Unexpected error:
      <exec.CodeExitError>: 
      command terminated with exit code 1
      {
          Err: <*errors.errorString | 0xc000c7c180>{
              s: "command terminated with exit code 1",
          },
          Code: 1,
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager_object_storage.go:361 @ 08/05/24 16:15:57.416
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func21.1({{0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0})
        github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager_object_storage.go:361 +0x4b47
    reflect.Value.call({0x25ea820?, 0xc000b821e0?, 0x13?}, {0x2d32c3a, 0x4}, {0xc000b2be18, 0x1, 0x1?})
        reflect/value.go:596 +0xca6
    reflect.Value.Call({0x25ea820?, 0xc000b821e0?, 0x35b0310?}, {0xc000b2be18?, 0x19bcd25?, 0xc00086de90?})
        reflect/value.go:380 +0xb9
rzetelskik commented 2 months ago

I already investigated it in https://github.com/scylladb/scylla-operator/issues/2005. It's a SM bug: https://github.com/scylladb/scylla-manager/issues/3873.

Leaving this as a tracker to avoid new duplicates.

rzetelskik commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2064/pull-scylla-operator-master-e2e-gke-parallel/1821155542387658752#1:test-build-log.txt%3A1202

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2077/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1824099494506008576#1:test-build-log.txt%3A1652

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2101/pull-scylla-operator-master-e2e-gke-parallel/1829078004123308032#1:test-build-log.txt%3A2499

rzetelskik commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2091/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1829168862705225728#1:test-build-log.txt%3A1442

rzetelskik commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2103/pull-scylla-operator-master-e2e-gke-parallel/1829180425826734080#1:test-build-log.txt%3A761

tnozicka commented 2 months ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2081/pull-scylla-operator-master-e2e-gke-parallel/1829501275406864384#1:test-build-log.txt%3A1247

rzetelskik commented 1 month ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1918/pull-scylla-operator-master-e2e-gke-parallel/1830909329918660608#1:test-build-log.txt%3A987

zimnx commented 1 month ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator-enterprise/31/pull-scylla-operator-enterprise-master-e2e-gke-parallel/1832025695719526400

zimnx commented 1 month ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator-enterprise/53/pull-scylla-operator-enterprise-master-e2e-gke-parallel/1833107801744871424

rzetelskik commented 3 weeks ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2142/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1841795382082801664#1:test-build-log.txt%3A1605

rzetelskik commented 3 weeks ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2143/pull-scylla-operator-master-e2e-gke-parallel/1842185562199429120#1:test-build-log.txt%3A1433

zimnx commented 1 week ago

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2160/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1848349953349390336