search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

roachtest: backup-restore/small-ranges failed [workload is too demanding] #130741

Open cockroach-teamcity opened 5 days ago

cockroach-teamcity commented 5 days ago

roachtest.backup-restore/small-ranges failed with artifacts on release-24.1 @ ab04f621a4c834c9f0cc7fd65187c531553ce384:

(test_runner.go:1247).runTest: test timed out (4h0m0s)
test artifacts and logs in: /artifacts/backup-restore/small-ranges/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-42198

msbutler commented 7 hours ago

I believe this timeout occured only because we got on unlucky. nothing stalled. All the backups ran in a reasonable time:

❯ grep "BACKUP" test.log
06:00:27 mixed_version_backup.go:1862: creating full backup via node 1: BACKUP DATABASE tpcc INTO 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/1_round-trip-test-backup_database-tpcc_s0f8?AUTH=implicit' AS OF SYSTEM TIME '1726293627212760142.0000000000' WITH detached
06:10:24 mixed_version_backup.go:1862: creating incremental backup via node 3: BACKUP DATABASE tpcc INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/1_round-trip-test-backup_database-tpcc_s0f8?AUTH=implicit' AS OF SYSTEM TIME '1726294224229566912.0000000000' WITH detached
06:43:53 mixed_version_backup.go:1862: creating full backup via node 3: BACKUP INTO 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/2_round-trip-test-backup_cluster_X0TJ?AUTH=implicit' AS OF SYSTEM TIME '1726296233994585331.0000000000' WITH detached, revision_history
06:59:31 mixed_version_backup.go:1862: creating incremental backup via node 1: BACKUP INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/2_round-trip-test-backup_cluster_X0TJ?AUTH=implicit' AS OF SYSTEM TIME '1726297171721394080.0000000000' WITH detached, revision_history
08:09:23 mixed_version_backup.go:1862: creating full backup via node 2: BACKUP INTO 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_ZNcR?AUTH=implicit' AS OF SYSTEM TIME '1726301363589046894.0000000000' WITH detached, revision_history, encryption_passphrase = 'Jq4FwTH97jlvYGIDgHi4w6IEAByGSCCZQnlhsEI'
08:22:13 mixed_version_backup.go:1862: creating incremental backup via node 2: BACKUP INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_ZNcR?AUTH=implicit' AS OF SYSTEM TIME '1726302133387978157.0000000000' WITH detached, revision_history, encryption_passphrase = 'Jq4FwTH97jlvYGIDgHi4w6IEAByGSCCZQnlhsEI'
08:28:01 mixed_version_backup.go:1862: creating incremental backup via node 2: BACKUP INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_ZNcR?AUTH=implicit' AS OF SYSTEM TIME '1726302481523421981.0000000000' WITH detached, revision_history, encryption_passphrase = 'Jq4FwTH97jlvYGIDgHi4w6IEAByGSCCZQnlhsEI'
08:28:35 mixed_version_backup.go:1862: creating incremental backup via node 1: BACKUP INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_ZNcR?AUTH=implicit' AS OF SYSTEM TIME '1726302515661297041.0000000000' WITH detached, revision_history, encryption_passphrase = 'Jq4FwTH97jlvYGIDgHi4w6IEAByGSCCZQnlhsEI'
09:36:07 mixed_version_backup.go:1862: creating full backup via node 2: BACKUP INTO 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/4_round-trip-test-backup_cluster_vIBs?AUTH=implicit' AS OF SYSTEM TIME '1726306567540285566.0000000000' WITH detached, encryption_passphrase = '4GJsw34R68vkFD8hPSVWt3gBSHtklaqiYP13mrKLstqT7'
09:47:43 mixed_version_backup.go:1862: creating incremental backup via node 1: BACKUP INTO LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/4_round-trip-test-backup_cluster_vIBs?AUTH=implicit' AS OF SYSTEM TIME '1726307263977779119.0000000000' WITH detached, encryption_passphrase = '4GJsw34R68vkFD8hPSVWt3gBSHtklaqiYP13mrKLstqT7'