search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

backupccl: failed to write metadata SST during TestTenantBackupNemesis #113852

Open stevendanna opened 1 year ago

stevendanna commented 1 year ago

Describe the problem

=== RUN   TestTenantBackupNemesis
    test_log_scope.go:170: test logs captured to: /tmp/_tmp/abc2733f63574a4c007456beb3ec251b/logTestTenantBackupNemesis1750470925
    test_log_scope.go:81: use -show-logs to present logs inline
    tenant_backup_nemesis_test.go:528: backup-nemesis: full backup started: BACKUP INTO $1 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:588: backup-nemesis: RENAME TENANT started
    tenant_backup_nemesis_test.go:379: no tenants to rename
    tenant_backup_nemesis_test.go:593: backup-nemesis: RENAME TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: DELETE TENANT started
    tenant_backup_nemesis_test.go:366: no tenants to delete
    tenant_backup_nemesis_test.go:593: backup-nemesis: DELETE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: DELETE TENANT started
    tenant_backup_nemesis_test.go:366: no tenants to delete
    tenant_backup_nemesis_test.go:593: backup-nemesis: DELETE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: REPLICATE TENANT started
    tenant_backup_nemesis_test.go:593: backup-nemesis: REPLICATE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: DELETE TENANT started
    tenant_backup_nemesis_test.go:593: backup-nemesis: DELETE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: CREATE UNIQUE INDEX (will fail) started
    tenant_backup_nemesis_test.go:532: backup-nemesis: full backup finished
    tenant_backup_nemesis_test.go:546: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1699264047155749000.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:593: backup-nemesis: CREATE UNIQUE INDEX (will fail) finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: CREATE INDEX started
    tenant_backup_nemesis_test.go:550: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:546: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1699264047310436000.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:550: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:546: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1699264047460111000.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:593: backup-nemesis: CREATE INDEX finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:593: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:593: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:588: backup-nemesis: CANCELED IMPORT INTO started
    tenant_backup_nemesis_test.go:550: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:546: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1699264047615010000.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:593: backup-nemesis: CANCELED IMPORT INTO finished
    tenant_backup_nemesis_test.go:248: 
                Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/backupccl/tenant_backup_nemesis_test.go:248
                Error:          Received unexpected error:
                                pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "span/\x12\xfe\x96\x00\x01\x12\xfe\x98\x00\x01"/0,0#0,SET, "span/\x12\xfe\x96\x00\x01\x12\xfe\x97\x00\x01"/1699264047.460111000,0#0,SET
                Test:           TestTenantBackupNemesis
    testutils.go:289: no Invalid Descriptors
    tenant_backup_nemesis_test.go:132: leaving /tmp/_tmp/abc2733f63574a4c007456beb3ec251b/TestTenantBackupNemesis3837983616 for inspection
    panic.go:523: -- test log scope end --
test logs left over in: /tmp/_tmp/abc2733f63574a4c007456beb3ec251b/logTestTenantBackupNemesis1750470925

Jira issue: CRDB-33228

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/disaster-recovery