search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.14k stars 3.81k forks source link

ccl/schemachangerccl: TestBackupSuccess_base_add_column failed #131583

Open github-actions[bot] opened 1 month ago

github-actions[bot] commented 1 month ago

ccl/schemachangerccl.TestBackupSuccess_base_add_column failed on master @ 8c52619671966cf15055f8b2e271ea82b59440c5:

                                  │    │    ├── ABSENT                → PUBLIC           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 2 (tbl_pkey+)}
                                  │    │    └── VALIDATED             → PUBLIC           ColumnNotNull:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 2 (tbl_pkey+)}
                                  │    ├── 4 elements transitioning toward TRANSIENT_ABSENT
                                  │    │    ├── TRANSIENT_DELETE_ONLY → TRANSIENT_ABSENT TemporaryIndex:{DescID: 118 (#118), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (tbl_pkey-)}
                                  │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 3}
                                  │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 3}
                                  │    │    └── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 3}
                                  │    ├── 2 elements transitioning toward ABSENT
                                  │    │    ├── PUBLIC                → VALIDATED        PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                  │    │    └── PUBLIC                → ABSENT           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 1 (tbl_pkey-)}
                                  │    └── 13 Mutation operations
                                  │         ├── MakePublicPrimaryIndexWriteOnly {"IndexID":1,"TableID":118}
                                  │         ├── SetIndexName {"IndexID":1,"Name":"crdb_internal_in...","TableID":118}
                                  │         ├── SetIndexName {"IndexID":2,"Name":"tbl_pkey","TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":3,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":3,"Kind":2,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":3,"IndexID":3,"Kind":2,"Ordinal":1,"TableID":118}
                                  │         ├── MakeValidatedColumnNotNullPublic {"ColumnID":3,"TableID":118}
                                  │         ├── MakeValidatedPrimaryIndexPublic {"IndexID":2,"TableID":118}
                                  │         ├── MakeIndexAbsent {"IndexID":3,"TableID":118}
                                  │         ├── MakeWriteOnlyColumnPublic {"ColumnID":3,"TableID":118}
                                  │         ├── RefreshStats {"TableID":118}
                                  │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
                                  │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"PostCommitNonRev..."}
                                  ├── Stage 2 of 3 in PostCommitNonRevertiblePhase
                                  │    ├── 3 elements transitioning toward ABSENT
                                  │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 1 (tbl_pkey-)}
                                  │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 1 (tbl_pkey-)}
                                  │    │    └── VALIDATED → DELETE_ONLY PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                  │    └── 5 Mutation operations
                                  │         ├── MakeWriteOnlyIndexDeleteOnly {"IndexID":1,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":1,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":1,"Kind":2,"TableID":118}
                                  │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
                                  │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"PostCommitNonRev..."}
                                  └── Stage 3 of 3 in PostCommitNonRevertiblePhase
                                       ├── 1 element transitioning toward TRANSIENT_ABSENT
                                       │    └── PUBLIC      → TRANSIENT_ABSENT IndexData:{DescID: 118 (#118), IndexID: 3}
                                       ├── 2 elements transitioning toward ABSENT
                                       │    ├── DELETE_ONLY → ABSENT           PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                       │    └── PUBLIC      → ABSENT           IndexData:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-)}
                                       └── 5 Mutation operations
                                            ├── MakeIndexAbsent {"IndexID":1,"TableID":118}
                                            ├── CreateGCJobForIndex {"IndexID":1,"TableID":118}
                                            ├── CreateGCJobForIndex {"IndexID":3,"TableID":118}
                                            ├── RemoveJobStateFromDescriptor {"DescriptorID":118,"JobID":1007973940837253100}
                                            └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"all stages compl..."}
        --- FAIL: TestBackupSuccess_base_add_column/post_commit_stage_7_of_7/restore_database (0.95s)
=== RUN   TestBackupSuccess_base_add_column/post_commit_stage_7_of_7
    --- FAIL: TestBackupSuccess_base_add_column/post_commit_stage_7_of_7 (6.95s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42619

rafiss commented 1 month ago 
=== RUN   TestBackupSuccess_base_add_column/post_commit_stage_7_of_7/restore_database
    backup.go:490: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/backup.go:490
            Error:          Not equal: 
                            expected: [][]string{[]string{"CREATE SCHEMA public"}, []string{"CREATE SEQUENCE public.sq1 MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 1 START 1"}, []string{"CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, j INT8 NOT NULL DEFAULT 42:::INT8, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"}}
                            actual  : [][]string{[]string{"CREATE SCHEMA public"}, []string{"CREATE SEQUENCE public.sq1 MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 1 START 1"}, []string{"CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"}}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -8,3 +8,3 @@
                              ([]string) (len=1) {
                            -  (string) (len=130) "CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, j INT8 NOT NULL DEFAULT 42:::INT8, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"
                            +  (string) (len=95) "CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"
                              }
            Test:           TestBackupSuccess_base_add_column/post_commit_stage_7_of_7/restore_database
            Messages:       backup contents:
                            parent_schema_name,object_name,object_type
                            NULL, public, schema
                            public, tbl, table
                            public, sq1, table

                            Schema change plan for ALTER TABLE ‹db›.public.‹tbl› ADD COLUMN ‹j› INT8 NOT NULL DEFAULT ‹42›;
                             ├── PostCommitPhase
                             │    └── Stage 1 of 1 in PostCommitPhase
                             │         ├── 2 elements transitioning toward PUBLIC
                             │         │    ├── WRITE_ONLY → VALIDATED PrimaryIndex:{DescID: 118 (#118), IndexID: 2 (tbl_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (tbl_pkey-)}
                             │         │    └── WRITE_ONLY → VALIDATED ColumnNotNull:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 2 (tbl_pkey+)}
                             │         └── 2 Validation operations
                             │              ├── ValidateIndex {"IndexID":2,"TableID":118}
                             │              └── ValidateColumnNotNull {"ColumnID":3,"IndexIDForValidation":2,"TableID":118}
                             └── PostCommitNonRevertiblePhase
                                  ├── Stage 1 of 3 in PostCommitNonRevertiblePhase
                                  │    ├── 4 elements transitioning toward PUBLIC
                                  │    │    ├── WRITE_ONLY            → PUBLIC           Column:{DescID: 118 (#118), ColumnID: 3 (j+)}
                                  │    │    ├── VALIDATED             → PUBLIC           PrimaryIndex:{DescID: 118 (#118), IndexID: 2 (tbl_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (tbl_pkey-)}
                                  │    │    ├── ABSENT                → PUBLIC           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 2 (tbl_pkey+)}
                                  │    │    └── VALIDATED             → PUBLIC           ColumnNotNull:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 2 (tbl_pkey+)}
                                  │    ├── 4 elements transitioning toward TRANSIENT_ABSENT
                                  │    │    ├── TRANSIENT_DELETE_ONLY → TRANSIENT_ABSENT TemporaryIndex:{DescID: 118 (#118), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (tbl_pkey-)}
                                  │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 3}
                                  │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 3}
                                  │    │    └── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 3}
                                  │    ├── 2 elements transitioning toward ABSENT
                                  │    │    ├── PUBLIC                → VALIDATED        PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                  │    │    └── PUBLIC                → ABSENT           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 1 (tbl_pkey-)}
                                  │    └── 13 Mutation operations
                                  │         ├── MakePublicPrimaryIndexWriteOnly {"IndexID":1,"TableID":118}
                                  │         ├── SetIndexName {"IndexID":1,"Name":"crdb_internal_in...","TableID":118}
                                  │         ├── SetIndexName {"IndexID":2,"Name":"tbl_pkey","TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":3,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":3,"Kind":2,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":3,"IndexID":3,"Kind":2,"Ordinal":1,"TableID":118}
                                  │         ├── MakeValidatedColumnNotNullPublic {"ColumnID":3,"TableID":118}
                                  │         ├── MakeValidatedPrimaryIndexPublic {"IndexID":2,"TableID":118}
                                  │         ├── MakeIndexAbsent {"IndexID":3,"TableID":118}
                                  │         ├── MakeWriteOnlyColumnPublic {"ColumnID":3,"TableID":118}
                                  │         ├── RefreshStats {"TableID":118}
                                  │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
                                  │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"PostCommitNonRev..."}
                                  ├── Stage 2 of 3 in PostCommitNonRevertiblePhase
                                  │    ├── 3 elements transitioning toward ABSENT
                                  │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 1 (tbl_pkey-)}
                                  │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 1 (tbl_pkey-)}
                                  │    │    └── VALIDATED → DELETE_ONLY PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                  │    └── 5 Mutation operations
                                  │         ├── MakeWriteOnlyIndexDeleteOnly {"IndexID":1,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":1,"TableID":118}
                                  │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":1,"Kind":2,"TableID":118}
                                  │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
                                  │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"PostCommitNonRev..."}
                                  └── Stage 3 of 3 in PostCommitNonRevertiblePhase
                                       ├── 1 element transitioning toward TRANSIENT_ABSENT
                                       │    └── PUBLIC      → TRANSIENT_ABSENT IndexData:{DescID: 118 (#118), IndexID: 3}
                                       ├── 2 elements transitioning toward ABSENT
                                       │    ├── DELETE_ONLY → ABSENT           PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
                                       │    └── PUBLIC      → ABSENT           IndexData:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-)}
                                       └── 5 Mutation operations
                                            ├── MakeIndexAbsent {"IndexID":1,"TableID":118}
                                            ├── CreateGCJobForIndex {"IndexID":1,"TableID":118}
                                            ├── CreateGCJobForIndex {"IndexID":3,"TableID":118}
                                            ├── RemoveJobStateFromDescriptor {"DescriptorID":118,"JobID":1007973940837253100}
                                            └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":1007973940837253100,"RunningStatus":"all stages compl..."}
        --- FAIL: TestBackupSuccess_base_add_column/post_commit_stage_7_of_7/restore_database (0.95s)
rafiss commented 1 month ago

@rimadeodhar I'm assigning to you for now as it seems the same as https://github.com/cockroachdb/cockroach/issues/129310, but if you have too many things on your plate, we can move this to someone else.

rafiss commented 3 weeks ago

I reproduced this on the 24.2 branch with this diff

diff --git a/pkg/sql/schemachanger/sctest/backup.go b/pkg/sql/schemachanger/sctest/backup.go
index 0d685a4ff7d..16191383c9a 100644
--- a/pkg/sql/schemachanger/sctest/backup.go
+++ b/pkg/sql/schemachanger/sctest/backup.go
@@ -33,7 +33,7 @@ import (
 // any time during a successful schema change.
 func BackupSuccess(t *testing.T, path string, factory TestServerFactory) {
    // These tests are expensive.
-   skip.UnderStress(t)
+   // skip.UnderStress(t)
    skip.UnderRace(t)
    skip.UnderDeadlock(t)

and command

./dev test pkg/ccl/schemachangerccl --stress -f='TestBackupSuccess_base_add_column$'

so this is a pre-existing issue, and not a GA-blocker.