roachtest: backup-restore/online-restore failed

[cockroach-teamcity]

roachtest.backup-restore/online-restore failed with artifacts on release-24.1.5-rc @ 520e3dfd703a240d803fea6dfa36f1d2c615c0f1:

(test_runner.go:1247).runTest: test timed out (4h0m0s)
test artifacts and logs in: /artifacts/backup-restore/online-restore/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #130778 roachtest: backup-restore/online-restore failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-master release-blocker] - #129187 roachtest: backup-restore/online-restore failed [relation "[239]" does not exist during schemachange workload] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-42270

[msbutler]

this restore command did not complete for 3 hours:

Running restore: RESTORE DATABASE schemachange FROM LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/4_round-trip-test-backup_database-schemachange_s0ro?AUTH=implicit'  WITH detached,     new_db_name = 'restore_4_round_trip_test_backup_database_schemachange_4', experimental deferred copy

... that's weird. link phase should take seconds to minutes

[msbutler]

this is a strange test flake. the restore job completed...

1004282330104823810 RESTORE RESTORE TABLE bank.bank FROM 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_table-bank.bank_cgRp/2024/09/17-060353.79?AUTH=implicit' WITH OPTIONS (into_db = 'restore_3_round_trip_test_backup_table_bank_bank_3', detached, experimental deferred copy)      roachprod   {221,232}   succeeded   NULL    2024-09-17 06:04:04.397009+00   2024-09-17 06:04:04.851904+00   2024-09-17 06:04:05.443381+00   2024-09-17 06:04:05.443413+00   1   NULL        2   8899453535071792554 2024-09-17 06:04:04.851904+00   2024-09-17 06:04:34.851904+00   1   NULL    NULL
1004282330104823811 RESTORE Background Data Download for RESTORE TABLE bank.bank FROM 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_table-bank.bank_cgRp/2024/09/17-060353.79?AUTH=implicit' WITH OPTIONS (into_db = 'restore_3_round_trip_test_backup_table_bank_bank_3', detached, experimental deferred copy)     roachprod   {}  succeeded   NULL    2024-09-17 06:04:05.406989+00   2024-09-17 06:04:06.017406+00   2024-09-17 06:04:07.17484+00    2024-09-17 06:04:07.174852+00   1   NULL        2   8662171652860580032 2024-09-17 06:04:06.017407+00   2024-09-17 06:04:36.017407+00   1   NULL    NULL