search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

schemachange: v24.1.2: nil pointer in backfillQueryIntoTable #127938

Open cockroach-sentry opened 3 months ago

cockroach-sentry commented 3 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5659470618/?referrer=webhooks_plugin

Panic Message:

panic.go:770: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:770
  | runtime.panicmem
  |     GOROOT/src/runtime/panic.go:261
  | runtime.sigpanic
  |     GOROOT/src/runtime/signal_unix.go:881
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).backfillQueryIntoTable.func2
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:341
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn.func1
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1794
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1881
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
  |     github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049
  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
  |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1103
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
  |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1066
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
  |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1041
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1868
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1795
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).backfillQueryIntoTable
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:331
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).refreshMaterializedView
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:304
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill
  |     github.com/cockroachdb/cockroach/pkg/sql/backfill.go:330
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2042
  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:939
  | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2722
  | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
  |     github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2883
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
  |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1598
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1599
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  |     github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:456
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
  |     github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:290
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1695
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:770: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/adopt.go#L289-L291](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/adopt.go#L289-L291) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/adopt.go#L455-L457](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/adopt.go#L455-L457) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/registry.go#L1598-L1600](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/registry.go#L1598-L1600) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/registry.go#L1597-L1599](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/jobs/registry.go#L1597-L1599) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2882-L2884](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2882-L2884) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2721-L2723](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2721-L2723) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L938-L940](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L938-L940) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2041-L2043](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L2041-L2043) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/backfill.go#L329-L331](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/backfill.go#L329-L331) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L303-L305](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L303-L305) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L330-L332](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L330-L332) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1794-L1796](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1794-L1796) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1867-L1869](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1867-L1869) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1040-L1042](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1040-L1042) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1065-L1067](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1065-L1067) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1102-L1104](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/db.go#L1102-L1104) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/txn.go#L1048-L1050](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/txn.go#L1048-L1050) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1880-L1882](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1880-L1882) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1793-L1795](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/internal.go#L1793-L1795) [https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L340-L342](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L340-L342) [GOROOT/src/runtime/signal_unix.go#L880-L882](GOROOT/src/runtime/signal_unix.go#L880-L882) [GOROOT/src/runtime/panic.go#L260-L262](GOROOT/src/runtime/panic.go#L260-L262) [GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771) 
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/jobs/adopt.go in pkg/jobs.(*Registry).resumeJob.func1 at line 290
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 456
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1599
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1598
pkg/sql/schema_changer.go in pkg/sql.schemaChangeResumer.Resume at line 2883
pkg/sql/schema_changer.go in pkg/sql.schemaChangeResumer.Resume.func1 at line 2722
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).exec at line 939
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).runStateMachineAndBackfill at line 2042
pkg/sql/backfill.go in pkg/sql.(*SchemaChanger).runBackfill at line 330
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).refreshMaterializedView at line 304
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).backfillQueryIntoTable at line 331
pkg/sql/internal.go in pkg/sql.(*InternalDB).Txn at line 1795
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn at line 1868
pkg/kv/db.go in pkg/kv.(*DB).Txn at line 1041
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 1066
pkg/kv/db.go in pkg/kv.runTxn at line 1103
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 1049
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn.func4 at line 1881
pkg/sql/internal.go in pkg/sql.(*InternalDB).Txn.func1 at line 1794
pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).backfillQueryIntoTable.func2 at line 341
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 881
GOROOT/src/runtime/panic.go in runtime.panicmem at line 261
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770

Tags

Tag Value
Command server
Environment v24.1.2
Go Version go1.22.3 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.1.2
Cockroach SHA 7e81be6de75205c3d08b0d8dcc6ca188306abc27
# of CPUs 18
# of Goroutines 1139

Jira issue: CRDB-40678

fqazi commented 6 days ago

This occurs because the sessionData was nil in the job: https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/sql/schema_changer.go#L341. We should add a guard rail and do a code inspection to see if we can spot any job creation code that never populates the session data