search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.87k stars 3.77k forks source link

[BugReport]Failed to print values by executing `cockroach debug keys path --values` #96148

Closed adzfolc closed 1 year ago

adzfolc commented 1 year ago

Describe the problem

Failed to print values by executing cockroach debug keys path --values

To Reproduce

1.Clone master branch, build and start local single server I'm now at commit * 69dd453d0e (HEAD -> master, upstream/master, origin/master, origin/HEAD) Merge #95658 2.Create a test table and insert some data 3.Execue command ./cockroach debug keys /Users/adzfolc/Workspace/crdbdata/26257 --values 4.Crash report

➜  cockroach git:(master) ./cockroach debug keys /Users/adzfolc/Workspace/crdbdata/26257 --values
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1  a panic has occurred!
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +close of closed channel
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +(1) attached stack trace
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  -- stack trace:
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.gopanic
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/panic.go:884
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:507
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.gopanic
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/panic.go:884
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble.Open.func1
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:111
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.gopanic
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/panic.go:890
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.closechan
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/chan.go:365
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).stopTicker
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:95
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).Close
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:108
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble/vfs.(*enospcFile).Close
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:355
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble.Open.func2
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:165
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/pebble.Open
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:207
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/storage.NewPebble
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1029
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/storage.Open
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/storage/open.go:239
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.OpenEngine
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/debug.go:173
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.runDebugKeys
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/debug.go:270
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:71
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/spf13/cobra.(*Command).execute
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/spf13/cobra.(*Command).ExecuteC
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/spf13/cobra.(*Command).Execute
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:143
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:68
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | main.main
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   main/pkg/cmd/cockroach/main.go:26
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.main
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/proc.go:250
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  | runtime.goexit
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +  |   GOROOT/src/runtime/asm_arm64.s:1172
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +Wraps: (2) close of closed channel
E230129 02:13:46.006928 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 1 +Error types: (1) *withstack.withStack (2) runtime.plainError
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2  a panic has occurred!
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +close of closed channel
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +(1) attached stack trace
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  -- stack trace:
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.gopanic
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/panic.go:884
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:507
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.gopanic
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/panic.go:884
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble.Open.func1
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:111
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.gopanic
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/panic.go:890
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.closechan
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/chan.go:365
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).stopTicker
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:95
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).Close
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:108
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble/vfs.(*enospcFile).Close
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:355
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble.Open.func2
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:165
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/pebble.Open
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:207
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/storage.NewPebble
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1029
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/storage.Open
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/storage/open.go:239
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli.OpenEngine
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/debug.go:173
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli.runDebugKeys
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/debug.go:270
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:71
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/spf13/cobra.(*Command).execute
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/spf13/cobra.(*Command).ExecuteC
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/spf13/cobra.(*Command).Execute
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:143
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   github.com/cockroachdb/cockroach/pkg/cli/cli.go:68
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | main.main
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   main/pkg/cmd/cockroach/main.go:26
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.main
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/proc.go:250
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  | runtime.goexit
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +  |   GOROOT/src/runtime/asm_arm64.s:1172
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +Wraps: (2) close of closed channel
E230129 02:13:46.007168 1 1@util/log/logcrash/crash_reporting.go:188  [T1] 2 +Error types: (1) *withstack.withStack (2) runtime.plainError
panic: close of closed channel [recovered]
    panic: close of closed channel [recovered]
    panic: close of closed channel [recovered]
    panic: close of closed channel [recovered]
    panic: close of closed channel

goroutine 1 [running]:
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.RecoverAndReportPanic({0x106e0fd48, 0x14000078090}, 0x140012923e8?)
    github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:131 +0x74
panic({0x1065c78e0, 0x106dc6bf0})
    GOROOT/src/runtime/panic.go:884 +0x204
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x140000021a0?, {0x106e0fd48, 0x14000078090})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x68
panic({0x1065c78e0, 0x106dc6bf0})
    GOROOT/src/runtime/panic.go:884 +0x204
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0x14000c51580, {0x106e0fd48, 0x14000078090})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:507 +0x1a0
panic({0x1065c78e0, 0x106dc6bf0})
    GOROOT/src/runtime/panic.go:884 +0x204
github.com/cockroachdb/pebble.Open.func1()
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:111 +0x158
panic({0x1065c78e0, 0x106dc6bf0})
    GOROOT/src/runtime/panic.go:890 +0x258
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).stopTicker(...)
    github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:95
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).Close(0x14001317dc0)
    github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:108 +0x28
github.com/cockroachdb/pebble/vfs.(*enospcFile).Close(0x14001292a48?)
    github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:355 +0x28
github.com/cockroachdb/pebble.Open.func2()
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:165 +0x40
github.com/cockroachdb/pebble.Open({0x16efdb757, 0x27}, 0x140004feb40)
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:207 +0xc38
github.com/cockroachdb/cockroach/pkg/storage.NewPebble({0x106e0fd48, 0x14000078090}, {{{{0x0, 0x0, 0x0}}, {0x16efdb757, 0x27}, 0x1, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1029 +0xf74
github.com/cockroachdb/cockroach/pkg/storage.Open({0x106e0fd48, 0x14000078090}, {{0x16efdb757?, 0x0?}, {0x106e72470?, 0x10aa8ae00?}}, 0x140014c2000, {0x14001293878, 0x4, 0x0?})
    github.com/cockroachdb/cockroach/pkg/storage/open.go:239 +0x238
github.com/cockroachdb/cockroach/pkg/cli.OpenEngine({0x16efdb757, 0x27}, 0x140009f9828?, {0x14001250a40, 0x2, 0x2})
    github.com/cockroachdb/cockroach/pkg/cli/debug.go:173 +0x134
github.com/cockroachdb/cockroach/pkg/cli.runDebugKeys(0x14001293a58?, {0x14000421160, 0x1, 0x13?})
    github.com/cockroachdb/cockroach/pkg/cli/debug.go:270 +0x120
github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1(0x10a90ee40?, {0x14000421160?, 0x1?, 0x2?})
    github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:71 +0x34
github.com/spf13/cobra.(*Command).execute(0x10a90ee40, {0x14000421140, 0x2, 0x2})
    github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856 +0x4dc
github.com/spf13/cobra.(*Command).ExecuteC(0x10a90bc40)
    github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974 +0x360
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
    github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
github.com/cockroachdb/cockroach/pkg/cli.doMain(0x10a90ee40, {0x1400007ab02, 0xa})
    github.com/cockroachdb/cockroach/pkg/cli/cli.go:143 +0x1cc
github.com/cockroachdb/cockroach/pkg/cli.Main()
    github.com/cockroachdb/cockroach/pkg/cli/cli.go:68 +0x194
main.main()
    main/pkg/cmd/cockroach/main.go:26 +0x1c

Expected behavior I expect that debug keys should be printed.

Additional data / screenshots 1.Dev Env Mac OS 13.2 (22D49) go version go1.19.5 darwin/arm64 2.Prepare for some test data

// execute in sql client
create table t(id int primary key);
insert into t select generate_series(1,100);
// execute in shell
./cockroach debug keys /Users/adzfolc/Workspace/crdbdata/26257 --values

3.Here is some log printed when server starts which may can help.

CockroachDB node starting at 2023-01-29 02:12:12.147435 +0000 UTC m=+3.559960251 (took 3.4s)
build:               CCL unknown @  (go1.19.5)
webui:               http://127.0.0.1:8081
sql:                 postgresql://root@127.0.0.1:26257/defaultdb?sslmode=disable
sql (JDBC):          jdbc:postgresql://127.0.0.1:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    /private/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/GoLand/___26257 <client cmd> --host=127.0.0.1:26257 --insecure
logs:                /Users/adzfolc/Workspace/crdbdata/26257/logs
temp dir:            /Users/adzfolc/Workspace/crdbdata/26257/cockroach-temp4055053780
external I/O path:   /Users/adzfolc/Workspace/crdbdata/26257/extern
store[0]:            path=/Users/adzfolc/Workspace/crdbdata/26257
storage engine:      pebble
clusterID:           6b89e5a4-50a7-4135-8f62-b7318ed72d33
status:              initialized new cluster
nodeID:              1

If applicable, add screenshots to help explain your problem.

Environment:

Additional context What was the impact?

Add any other context about the problem here.

Jira issue: CRDB-23966

blathers-crl[bot] commented 1 year ago

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

If we have not gotten back to your issue within a few business days, you can try the following:

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

adzfolc commented 1 year ago

After re-compile server by executing ./dev build cockroach via bazel, I can print debug keys successfully now. ps: I used to compile server by executing make build, which can't print debug keys successfully.

yuzefovich commented 1 year ago

make is deprecated, dev is the way to build cockroach nowadays.

adzfolc commented 1 year ago

@yuzefovich Hi,I'm not familiar with Bazel, and while I pull latest code from master, I can't debug crdb now with goland.Is there any toturial about how to debug crdb builded by bazel in goland?

./cockroach version
Build Tag:        v23.1.0-alpha.1-1527-gdc32242213-dirty
Build Time:       2023/02/02 02:48:02
Distribution:     CCL
Platform:         darwin arm64 (arm64-apple-darwin22.3.0)
Go Version:       go1.19.4
C Compiler:       Apple LLVM 14.0.0 (clang-1400.0.29.202)
Build Commit ID:  dc3224221324dcc4336861232003f2ec1aac4f9c
Build Type:       development

And following is my goland bazel conf.Is there anything wrong with my conf? 图片

yuzefovich commented 1 year ago

Try ./dev gen go - this should generate some of the code and put it where Goland expects it.

adzfolc commented 1 year ago

Thx for reply, I've executed several cmds as ./dev -h, like ./dev gen bazel, ./dev gen go. And, truly I can start crdb server via goland, and bazel conf is like screen shoot like above. The run banner is grey, and I can't activate it, while run banner is green, and I can run crdb server in goland. 图片 Can u help me with how to debug crdb in goland?

yuzefovich commented 1 year ago

I'm assuming that you're trying to run CRDB via a debugger in GoLand, right? Here is an example of how I run a single node cockroach cluster via the debugger: Screenshot 2023-02-01 at 7 41 00 PM Note that it's not running a "bazel" command - it's a "regular" go run. (This is after ./dev gen go.)

adzfolc commented 1 year ago

I'm assuming that you're trying to run CRDB via a debugger in GoLand, right? Here is an example of how I run a single node cockroach cluster via the debugger: Screenshot 2023-02-01 at 7 41 00 PM Note that it's not running a "bazel" command - it's a "regular" go run. (This is after ./dev gen go.)

After configure as above, I push run banner, and meet another problem,

# github.com/knz/strtime
gmtime_r.c:47:1: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
# github.com/cockroachdb/cockroach/pkg/cmd/cockroach
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x105205a08?, 0x1401bb78150?, 0x1400071c180, {0x14015704820?, 0x1?}, {0x14027bc4000, 0x53})
    /opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000148000)
    /opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x10513c842, 0x14}, {0x1051401ad, ...}, ...})
    /opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
    /opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/main.go:72 +0xd98

Compilation finished with exit code 2

图片

➜  cockroach git:(master) ✗ go version
go version go1.19.5 darwin/arm64
➜  cockroach git:(master) ✗ ./cockroach version
Build Tag:        v23.1.0-alpha.1-1527-gdc32242213-dirty
Build Time:       2023/02/02 02:48:02
Distribution:     CCL
Platform:         darwin arm64 (arm64-apple-darwin22.3.0)
Go Version:       go1.19.4
C Compiler:       Apple LLVM 14.0.0 (clang-1400.0.29.202)
Build Commit ID:  dc3224221324dcc4336861232003f2ec1aac4f9c
Build Type:       development

My go version is go1.19.5, and ./cockroach version shows go version is go1.19.4.Is there any relationship between error above and go version diff?

yuzefovich commented 1 year ago

Huh, I've never seen an error like this before. It might be related to go version so consider downgrading to 1.19.4. Also, consider running rm -rf vendor if you have ./vendor folder (we recently removed the old way we vendored dependencies, and the old folder might be interfering with the build although it seems unlikely). Otherwise, I'm out of ideas.

adzfolc commented 1 year ago

Huh, I've never seen an error like this before. It might be related to go version so consider downgrading to 1.19.4. Also, consider running rm -rf vendor if you have ./vendor folder (we recently removed the old way we vendored dependencies, and the old folder might be interfering with the build although it seems unlikely). Otherwise, I'm out of ideas.

I think I'm close to the end of the problem. Now, I've done things as following:

  1. Downgrade go to 1.19.4 to keep pace with crdb 
    ➜  ~ go version
go version go1.19.4 darwin/arm64
➜  cockroach git:(master) ✗ ./cockroach version
Build Tag:        v23.1.0-alpha.1-1527-gdc32242213
Build Time:       2023/02/02 06:09:47
Distribution:     CCL
Platform:         darwin arm64 (arm64-apple-darwin22.3.0)
Go Version:       go1.19.4
C Compiler:       Apple LLVM 14.0.0 (clang-1400.0.29.202)
Build Commit ID:  dc3224221324dcc4336861232003f2ec1aac4f9c
Build Type:       development
  2. I can start crdb server in zsh shell. 
    ➜  cockroach git:(master) ✗ ./cockroach-short start-single-node --listen-addr=127.0.0.1:26257 --http-addr=127.0.0.1:8081 --store /Users/adzfolc/Workspace/crdbdata --insecure
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
* 
* This mode is intended for non-production testing only.
* 
* In this mode:
* - Your cluster is open to any client that can access 127.0.0.1.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
* 
* - https://go.crdb.dev/issue-v/53404/v23.1
* - https://www.cockroachlabs.com/docs/v23.1/secure-a-cluster.html
*
*
* WARNING: Running a server without --sql-addr, with a combined RPC/SQL listener, is deprecated.
* This feature will be removed in the next version of CockroachDB.
*
CockroachDB node starting at 2023-02-02 07:41:00.234675 +0000 UTC m=+1.422750626 (took 1.2s)
build:               CCL v23.1.0-alpha.1-1527-gdc32242213 @ 2023/02/02 06:09:47 (go1.19.4)
webui:               http://127.0.0.1:8081
sql:                 postgresql://root@127.0.0.1:26257/defaultdb?sslmode=disable
sql (JDBC):          jdbc:postgresql://127.0.0.1:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    ./cockroach-short <client cmd> --host=127.0.0.1:26257 --insecure
logs:                /Users/adzfolc/Workspace/crdbdata/logs
temp dir:            /Users/adzfolc/Workspace/crdbdata/cockroach-temp4084431707
external I/O path:   /Users/adzfolc/Workspace/crdbdata/extern
store[0]:            path=/Users/adzfolc/Workspace/crdbdata
storage engine:      pebble
clusterID:           4413793c-7314-486c-98c6-8253b2bb6ac9
status:              restarted pre-existing node
nodeID:              1
^CNote: a second interrupt will skip graceful shutdown and terminate forcefully
initiating graceful shutdown of server
server drained and shutdown completed
*
* INFO: interrupted
*
ERROR: interrupted
Failed running "start-single-node"

However, in GoLand, I start server via Go Build, conf is like ur screen shoot. And, when I debug, server starts failed:

GOROOT=/usr/local/go #gosetup
GOPATH=/Users/adzfolc/go #gosetup
/usr/local/go/bin/go build -o /private/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/GoLand/___1crdb -gcflags all=-N -l . #gosetup
# github.com/knz/strtime
gmtime_r.c:47:1: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
# github.com/cockroachdb/cockroach/pkg/cmd/cockroach
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x104bc9a08?, 0x140000a6300?, 0x14000a14080, {0x1400b705220?, 0x1?}, {0x14010c90c60, 0x53})
    /usr/local/go/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000104000)
    /usr/local/go/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x104b00802, 0x14}, {0x104b0416d, ...}, ...})
    /usr/local/go/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
    /usr/local/go/src/cmd/link/main.go:72 +0xd98

Compilation finished with exit code 2

Err comes from code as below 图片

yuzefovich commented 1 year ago

What hardware and OS are you running on?

adzfolc commented 1 year ago

HardWare: arm64 CPU: Apple M1 Max OS: macOS 13.2 (22D49)

adzfolc commented 1 year ago

I still could debug crdb via goland about 1 week ago(at Jan 29, still with Makefile), and after pulling code(about several days ago) from switching to bazel, I can't debug now.

yuzefovich commented 1 year ago

Not sure what the issue is, I'll ask my colleagues to look into this.

adzfolc commented 1 year ago

In addition, I'm great interested in db, and I'm familiar with sql parser, analyzer; txn xa protocol and know sth about consensus algo(know paxos and new to raft).Recently, I'm learning crdb cdc(rangefeed) and sth about distributed layer(meta 1/2 and ranges merge/split). Can I get enrolled in crdb development from some trival problems? I'm not sure how to contact you is reasonable, I'm available at mail(adzfolc@gmail.com) and CockroachDB Community Slack(username adzfolc). And I'm very very willing to participate in crdb contribution as a community member. Wish U A Goo Day! LOL~ :D

yuzefovich commented 1 year ago

One my colleagues suggested that it must be a bug in Go linker, so you should file an issue upstream with the reproduction steps.

In terms of contributing, take a look at #41815 for some guidance.

rickystewart commented 1 year ago

Putting the content build --force_build_cdeps in a .bazelrc.user file in your checkout and then re-running ./dev gen go may fix the problem, I am not sure.

adzfolc commented 1 year ago

One my colleagues suggested that it must be a bug in Go linker, so you should file an issue upstream with the reproduction steps.

In terms of contributing, take a look at #41815 for some guidance.

I've report this proble to go upstream. And issue to track is linked to https://github.com/golang/go/issues/58330

adzfolc commented 1 year ago

Putting the content build --force_build_cdeps in a .bazelrc.user file in your checkout and then re-running ./dev gen go may fix the problem, I am not sure.

I tried and failed. I guess that build --force_build_cdeps has same effect like ./dev gen cgo?

adzfolc commented 1 year ago

After re-init my Mac for several times, and re-install dev chain, finally I can debug now.Thanks for your help!