search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.97k stars 3.79k forks source link

demo: panic with workload init #94654

Open dbist opened 1 year ago

dbist commented 1 year ago

Describe the problem

Please describe the issue you observed, and any steps we can take to reproduce it:

To Reproduce

cockroach demo --nodes 9 --with-load

\demo add region=us-central1,az=a
\demo add region=us-central1,az=b
\demo add region=us-central1,az=c

\demo shutdown 7
\demo shutdown 8
\demo shutdown 9

# capture a node to connect to
\demo ls

in another window

cockroach workload init tpcc --db movr "postgresql://demo:demo55320@127.0.0.1:26268/movr?sslmode=require&sslrootcert=%2Fvar%2Ffolders%2F1c%2Fj4z3c4gn5gx3mftgpcmfcrsc0000gq%2FT%2Fdemo4578662%2Fca.crt"

init will fail with the following message

Error: importing fixture: importing table history: pq: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101533696 currently allocated, 134217728 bytes in budget

now attempt to exit out of the original cockroach demo terminal

demo@127.0.0.1:26257/movr> \q
*
* ERROR: a panic has occurred!
* pebble: closed
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/pebble.(*DB).Apply
*   |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:702
*   | github.com/cockroachdb/pebble.(*Batch).Commit
*   |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1049
*   | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
*   |   github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:545
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:983
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:664
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:641
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:308
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
* Wraps: (2) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/pebble.(*DB).Close
*   |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:1239
*   | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).Close
*   |   github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1068
*   | github.com/cockroachdb/cockroach/pkg/server.(*stickyInMemEnginesRegistryImpl).deleteEngine
*   |   github.com/cockroachdb/cockroach/pkg/server/sticky_engine.go:186
*   | github.com/cockroachdb/cockroach/pkg/server.(*stickyInMemEnginesRegistryImpl).CloseAllStickyInMemEngines
*   |   github.com/cockroachdb/cockroach/pkg/server/sticky_engine.go:176
*   | github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Start.func3.1
*   |   github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:286
*   | github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558
*   | github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Close
*   |   github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:811
*   | github.com/cockroachdb/cockroach/pkg/cli.runDemoInternal
*   |   github.com/cockroachdb/cockroach/pkg/cli/demo.go:372
*   | github.com/cockroachdb/cockroach/pkg/cli.runDemo
*   |   github.com/cockroachdb/cockroach/pkg/cli/demo.go:210
*   | github.com/cockroachdb/cockroach/pkg/cli.init.5.func1
*   |   github.com/cockroachdb/cockroach/pkg/cli/demo.go:59
*   | github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1
*   |   github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:71
*   | github.com/spf13/cobra.(*Command).execute
*   |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856
*   | github.com/spf13/cobra.(*Command).ExecuteC
*   |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
*   | github.com/spf13/cobra.(*Command).Execute
*   |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
*   | github.com/cockroachdb/cockroach/pkg/cli.Run
*   |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
*   | github.com/cockroachdb/cockroach/pkg/cli.doMain
*   |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:143
*   | github.com/cockroachdb/cockroach/pkg/cli.Main
*   |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:68
*   | main.main
*   |   main/pkg/cmd/cockroach/main.go:26
*   | [...repeated from below...]
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/pebble.init
*   |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:44
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6340
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.doInit
*   |   GOROOT/src/runtime/proc.go:6317
*   | runtime.main
*   |   GOROOT/src/runtime/proc.go:233
*   | runtime.goexit
*   |   src/runtime/asm_arm64.s:1172
* Wraps: (4) pebble: closed
* Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: Queued as error 8638dbe8c3ff487892d2381217d33e02
*
panic: pebble: closed [recovered]
        panic: pebble: closed

goroutine 46696 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x140075b97a0, {0x1098d1060, 0x14011ae4a80})
        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:248 +0xa0
panic({0x109495c60, 0x1402bccafa8})
        GOROOT/src/runtime/panic.go:884 +0x204
github.com/cockroachdb/pebble.(*DB).Apply(0x140143b69b8?, 0x105ab08dc?, 0x140143b69b8?)
        github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:702 +0x378
github.com/cockroachdb/pebble.(*Batch).Commit(...)
        github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1049
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit(0x14034a6d000?, 0x60?)
        github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:545 +0x4c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:983 +0x10a8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:664 +0x138
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x14018616000, 0x14018711740?)
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:641 +0x88
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0x14001ba43c0, {0x1098d1060, 0x14011ae4a80})
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:308 +0x258
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x358

Expected behavior

the system should prevent from getting in this state or fail gracefully.

Environment:

Additional context

minor impact

Jira issue: CRDB-23067

knz commented 1 year ago

cc @rafiss for triage. I suspect mistaken assumption about shutdown order. Maybe that will go to the storage team.

herkolategan commented 1 year ago

This microbenchmark failure could possibly be related #96013 and is an easy way to reproduce if it is.