search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed #122176

Closed cockroach-teamcity closed 5 months ago

cockroach-teamcity commented 5 months ago

pkg/ccl/backupccl/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips failed on release-24.1 @ 592a66827e1864303791b4f3940fb52331bc949d:

=== RUN   TestBackupRestoreRandomDataRoundtrips
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreRandomDataRoundtrips3255197590
    test_log_scope.go:81: use -show-logs to present logs inline
    jobs_verification.go:98: job failed: failed to generate and send download spans: failed to download spans on 1 nodes; n1 returned error requesting download spans from node 1 (NODE_STATUS_LIVE): rpc error: code = Unknown desc = CopySpan cannot copy empty span \x92\x00#10423,DELSIZED \x93\x00#72057594037927935,RANGEDEL
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreRandomDataRoundtrips3255197590
--- FAIL: TestBackupRestoreRandomDataRoundtrips (30.68s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-37743

msbutler commented 5 months ago

@RaduBerinde We're still observing empty copy spans on 24.1. Is this a new bug?

RaduBerinde commented 5 months ago

Yeah, all known bugs have been fixed, so we must have missed something.. We are sure that the test never causes external ingestions where there are no keys in the span?

RaduBerinde commented 5 months ago

@msbutler I have found that the test attempts an empty ingestion. I ran under stress against https://github.com/cockroachdb/pebble/pull/3506 and I got:

* panic: external ingestion is empty: data/959416941262372865.sst ("\xf6w\x8a\x00" "\xf6w\x8b\x00" inclusive=false)

This means that there are no keys in that sst within the bound.

RaduBerinde commented 5 months ago

Here is an LSM that triggers the error. Tables 40 and 7 have no keys inside.

msbutler commented 5 months ago

aha! thanks for digging into this. I'll look into why this could happen.

msbutler commented 5 months ago

I stressed this test on top of a patch that panics in backup if it creates a backup file without any point keys, and no panic occurred during the repro, suggesting the source of the empty download span is not due to backup.

RaduBerinde commented 5 months ago

The physical file can have keys, the question is whether the portion that is being ingested (StartKey to EndKey) has any keys.

msbutler commented 5 months ago

Ah, to clarify: in DR land, a physical backup sst is partitioned into several backup files (i.e. roughly the export request responses). In Online restore, we create virtual sst's for each backup file. The terminology is really confusing and we should probably come up with a new term for backup file.

My patch above was verifying that no backup file was empty during the test failure.

RaduBerinde commented 5 months ago

The panic above shows that there is an external ingestion with no keys between the bounds. OR is the only one that does external ingestions. Maybe the bounds get screwed up somehow?

RaduBerinde commented 5 months ago

I can extend the patch to show all the keys in the physical file, that should give us some clue.

msbutler commented 5 months ago

Ah, that LSM visualizer is really helpful. I think understand how this could happen. Suppose table 120 has a secondary index, restore will partition the table's key space into two units of work (a restoreSpanEntry): [Table/120/1-Table/120/2),[Table/120/2-Table/120/3), but what if that second index is empty?

On the backup side, we very well could create a non empty backup file that spans the two indexes: [Table/120/1-Table/120/3) -- we construct backup units of work on the table level-- but restore we create units of work on the table-index level and will break up the table wide span for virtual sst ingestion here. Given all of this, we'll create a virtual sst that has no keys: [Table/120/2-Table/120/3).

To reproduce, I'll try to backup and online restore a table with an empty secondary index.

msbutler commented 5 months ago

here's the schema of the backed up table during a failing test case. i wonder if we can induce this via a partial index:

I240412 14:25:02.267796 906 5@util/log/event_log.go:32 â<8b>® [T1,Vsystem,n1,client=127.0.0.1:41058,hostssl,user=root] 127 ={"Timestamp":1712931902263077505,"EventType":"create_table","Statement":"CREATE TABLE â<80>¹randâ<80>º.public.â<80>¹t    able_1â<80>º (â<80>¹col1_0â<80>º BOX2D NULL, â<80>¹\"col1|_1\"â<80>º BOOL NOT NULL, â<80>¹col1_2â<80>º REGCLASS NULL, â<80>¹col1_3â<80>º JSONB NOT NULL, â<80>¹col1_4â<80>º VARCHAR NOT NULL, PRIMARY KEY (â<80>¹col1_4â<80>º, â<80>¹\"col1|_1\"â    <80>º ASC), INDEX (â<80>¹col1_0â<80>º ASC), INDEX (â<80>¹crdb_internal_idx_exprâ<80>º DESC), INDEX (â<80>¹col1_3â<80>º DESC) STORING (â<80>¹col1_0â<80>º), UNIQUE (â<80>¹\"col1|_1\"â<80>º, â<80>¹col1_4â<80>º DESC) STORING (â<80>¹col1_0â<80>º,     â<80>¹col1_3â<80>º) WHERE â<80>¹table_1â<80>º.â<80>¹\"col1|_1\"â<80>º, INDEX (â<80>¹\"col1|_1\"â<80>º) WHERE â<80>¹table_1â<80>º.â<80>¹col1_4â<80>º >= â<80>¹'\"'â<80>º:::STRING)","Tag":"CREATE TABLE","User":"root","DescriptorID":106,"TableN    ame":"â<80>¹rand.public.table_1â<80>º"}
msbutler commented 5 months ago

Alrighty, I think my above theory holds: if I patch restore to create units of work on the table level, I cannot reproduce the empty download span bug. Even better, if i patch backup to create work on the table-index level, I cannot hit the bug either.

I still haven't figured out how to easily repro this bug. For example, I cant get this unit test to fail under stess, even though it contains an empty index. Some how in mvcc export request, we generally avoid returning an export response that spans the empty index span.

I also am unsure how we should fix this. Some candidates:

cockroach-teamcity commented 5 months ago

pkg/ccl/backupccl/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips failed on release-24.1 @ 53fcac10e573f8b3f0f6729fb7840f5ffba400b3:

=== RUN   TestBackupRestoreRandomDataRoundtrips
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreRandomDataRoundtrips2167932939
    test_log_scope.go:81: use -show-logs to present logs inline
    jobs_verification.go:98: job failed: failed to generate and send download spans: failed to download spans on 1 nodes; n1 returned error requesting download spans from node 1 (NODE_STATUS_LIVE): rpc error: code = Unknown desc = CopySpan cannot copy empty span \x91\x00#10035,DELSIZED \x92\x00#72057594037927935,RANGEDEL
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreRandomDataRoundtrips2167932939
--- FAIL: TestBackupRestoreRandomDataRoundtrips (29.79s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #122341 pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed [C-test-failure O-robot P-0 T-disaster-recovery branch-master release-blocker]

This test on roachdash | Improve this report!

msbutler commented 5 months ago

closed via https://github.com/cockroachdb/cockroach/pull/122679