crash with GODEBUG=cgocheck=2: Go pointer stored into non-Go memory

davepacheco commented 1 year ago

Describe the problem

While trying to track down #82958 and a raft of similar problems, I stumbled on GODEBUG=cgocheck=2 and I found that cockroach seems to crash with this set. Note that while most of my issues have been on illumos, this is easily reproduced for me on Linux.

I know the code thinks its behavior is safe despite violating Go's rules here (which would make this a false positive, I guess). I'm not sure that's true -- see below.

To Reproduce

I set up an AWS VM using ami-017fecd1353bcc96e (Ubuntu 22.04 LTS, amd64).
I downloaded CockroachDB from https://binaries.cockroachdb.com.
I started a single-node cluster with GODEBUG=cgocheck=2 set and waited for it to appear to come up. Then I ^C'd it:

$ GODEBUG=cgocheck=2 cockroach start-single-node --insecure --store=path=$PWD/cockroachdb-store 
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
* 
* This mode is intended for non-production testing only.
* 
* In this mode:
* - Your cluster is open to any client that can access any of your IP addresses.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
* 
* - https://go.crdb.dev/issue-v/53404/v22.1
* - https://www.cockroachlabs.com/docs/v22.1/secure-a-cluster.html
*
*
* WARNING: neither --listen-addr nor --advertise-addr was specified.
* The server will advertise "ip-172-31-49-171" to other nodes, is this routable?
* 
* Consider using:
* - for local-only servers:  --listen-addr=localhost
* - for multi-node clusters: --advertise-addr=<host/IP addr>
* 
*
^CNote: a second interrupt will skip graceful shutdown and terminate forcefully
initiating graceful shutdown of server
too early to drain; used hard shutdown instead
*
* INFO: interrupted
*
ERROR: interrupted
Failed running "start-single-node"

Then I started the single-node cluster again with the same arguments. This time it quickly bails out:

$ GODEBUG=cgocheck=2 cockroach start-single-node --insecure --store=path=$PWD/cockroachdb-store 
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
* 
* This mode is intended for non-production testing only.
* 
* In this mode:
* - Your cluster is open to any client that can access any of your IP addresses.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
* 
* - https://go.crdb.dev/issue-v/53404/v22.1
* - https://www.cockroachlabs.com/docs/v22.1/secure-a-cluster.html
*
$

Here's the stderr:

``` $ cat cockroachdb-store/logs/cockroach-stderr.log I221122 22:49:41.609172 1 util/log/file_sync_buffer.go:238 ⋮ [config] file created at: 2022/11/22 22:49:41 I221122 22:49:41.609221 1 util/log/file_sync_buffer.go:238 ⋮ [config] running on machine: ‹ip-172-31-49-171› I221122 22:49:41.609234 1 util/log/file_sync_buffer.go:238 ⋮ [config] binary: CockroachDB CCL v22.1.9 (x86_64-pc-linux-gnu, built 2022/10/17 18:24:57, go1.17.11) I221122 22:49:41.609243 1 util/log/file_sync_buffer.go:238 ⋮ [config] arguments: [‹cockroach› ‹start-single-node› ‹--insecure› ‹--store=path=/home/ubuntu/cockroachdb-store›] I221122 22:49:41.609263 1 util/log/file_sync_buffer.go:238 ⋮ [config] log format (utf8=✓): crdb-v2 I221122 22:49:41.609271 1 util/log/file_sync_buffer.go:238 ⋮ [config] line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid [chan@]file:line redactionmark \[tags\] [counter] msg I221122 22:49:41.608887 1 util/log/flags.go:198 [-] 1 stderr capture started fatal error: Go pointer stored into non-Go memory goroutine 153 [running]: runtime.throw({0x4f75f26, 0xc001bc6188}) GOROOT/src/runtime/panic.go:1198 +0x71 fp=0xc001bc12f8 sp=0xc001bc12c8 pc=0x48ac91 runtime.cgoCheckBits(0x4c166a, 0xc0010ca9c0, 0xc000089800, 0xc001bc13e0) GOROOT/src/runtime/cgocheck.go:192 +0xc5 fp=0xc001bc1348 sp=0xc001bc12f8 pc=0x458b85 runtime.cgoCheckTypedBlock(0x7f6ab2614000, 0x51bf6e8, 0xc001bc1410, 0x4bf487) GOROOT/src/runtime/cgocheck.go:118 +0x57 fp=0xc001bc13d0 sp=0xc001bc1348 pc=0x4587b7 runtime.cgoCheckMemmove(0x4d54760, 0x4889760, 0xc000673020, 0x7f6ab2614000, 0xc000673020) GOROOT/src/runtime/cgocheck.go:76 +0x6c fp=0xc001bc1420 sp=0xc001bc13d0 pc=0x45866c runtime.typedmemmove(0x4d54760, 0x7f6ab2614000, 0xc001bc1490) GOROOT/src/runtime/mbarrier.go:173 +0x87 fp=0xc001bc1458 sp=0xc001bc1420 pc=0x4677e7 github.com/cockroachdb/pebble/internal/cache.newEntry(0xc0012cb450, {{0x7f6ab2613000, 0xc001bc15e0}, 0x11a4afd}, 0x53) github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/entry.go:68 +0x108 fp=0xc001bc1500 sp=0xc001bc1458 pc=0x1186aa8 github.com/cockroachdb/pebble/internal/cache.(*shard).Set(0xc0012cb450, 0xffffffffffffffff, 0xc001bc15e0, 0x1146bd4, 0x7f6ab2613000) github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go:148 +0x265 fp=0xc001bc1598 sp=0xc001bc1500 pc=0x11842a5 github.com/cockroachdb/pebble/internal/cache.(*Cache).Set(0xc000cf0440, 0x7f6ab2613020, 0x7f6ab2613020, 0x540eea, 0x4c166a) github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go:750 +0x46 fp=0xc001bc15f0 sp=0xc001bc1598 pc=0x1186186 github.com/cockroachdb/pebble/sstable.(*Reader).readBlock(0xc001bc4000, {0xc0004241f8, 0x925fc48}, 0x0, 0x0) github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:2364 +0x637 fp=0xc001bc1708 sp=0xc001bc15f0 pc=0x11a5357 github.com/cockroachdb/pebble/sstable.(*Reader).readMetaindex(0xc001bc4000, {0x0, 0x7f6ab993c1b8}) github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:2411 +0x5c fp=0xc001bc1ab0 sp=0xc001bc1708 pc=0x11a5afc github.com/cockroachdb/pebble/sstable.NewReader({0x7f6ab993c1b8, 0xc000cf0440}, {0xc001b2acd0, 0x91a3ea0, 0xc0008ea750, {0x4f2be20, 0x18}}, {0xc001bc1d18, 0x3, 0x3}) github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:2792 +0x3bc fp=0xc001bc1c30 sp=0xc001bc1ab0 pc=0x11a8a7c github.com/cockroachdb/pebble.(*tableCacheValue).load(0xc0005fc1c0, 0xc000b7e420, 0xc0017d1400, 0xc0003ef508) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:848 +0x2f9 fp=0xc001bc1d60 sp=0xc001bc1c30 pc=0x1231b99 github.com/cockroachdb/pebble.(*tableCacheShard).findNode.func2({0x63c3e78, 0xc00046a300}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:618 +0x29 fp=0xc001bc1d90 sp=0xc001bc1d60 pc=0x1230bc9 runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0x0}}, 0xc001bc1e80) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 fp=0xc001bc1e00 sp=0xc001bc1d90 pc=0x104b0e3 github.com/cockroachdb/pebble.(*tableCacheShard).findNode(0xc0017d1400, 0xc000b7e420, 0xc0003ef508) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:617 +0x485 fp=0xc001bc1eb8 sp=0xc001bc1e00 pc=0x1230b45 github.com/cockroachdb/pebble.(*tableCacheShard).newIters(0xc0017d1400, 0x0, 0x0, 0xc000df68d8, 0xc0003ef508) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:340 +0x7d fp=0xc001bc1fc0 sp=0xc001bc1eb8 pc=0x122f35d github.com/cockroachdb/pebble.(*tableCacheContainer).newIters(0x0, 0x0, 0x0, 0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:117 +0x45 fp=0xc001bc1ff8 sp=0xc001bc1fc0 pc=0x122e565 github.com/cockroachdb/pebble.(*tableCacheContainer).newIters-fm(0xc001bc2a38, 0x0, 0xffff) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:114 +0x32 fp=0xc001bc2028 sp=0xc001bc1ff8 pc=0x123beb2 github.com/cockroachdb/pebble.(*compaction).newInputIter(0xc000df6840, 0xc0009d0120) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1081 +0xed8 fp=0xc001bc2d18 sp=0xc001bc2028 pc=0x11d3c78 github.com/cockroachdb/pebble.(*DB).runCompaction(0xc0001fe480, 0x3, 0xc000df6840, {0x6320180, 0x9befd40}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2024 +0xd99 fp=0xc001bc3be0 sp=0xc001bc2d18 pc=0x11dc459 github.com/cockroachdb/pebble.(*DB).compact1(0xc0001fe480, 0xc000df6840, 0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1889 +0x2de fp=0xc001bc3e80 sp=0xc001bc3be0 pc=0x11daf3e github.com/cockroachdb/pebble.(*DB).compact.func1({0x63c3e78, 0xc00046a0f0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1850 +0xad fp=0xc001bc3ee8 sp=0xc001bc3e80 pc=0x11dab6d runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c700, 0xf0e9dcce0e33934, 0x7f65976d5f5af04e}}, 0xc001289788) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 fp=0xc001bc3f58 sp=0xc001bc3ee8 pc=0x104b0e3 github.com/cockroachdb/pebble.(*DB).compact(0xfbaf5016fa3faee3, 0x59df6b6069f4fcfb, 0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1847 +0x6b fp=0xc001bc3fb8 sp=0xc001bc3f58 pc=0x11daa8b github.com/cockroachdb/pebble.(*DB).maybeScheduleCompactionPicker·dwrap·13() github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1641 +0x2e fp=0xc001bc3fe0 sp=0xc001bc3fb8 pc=0x11d93ae runtime.goexit() GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc001bc3fe8 sp=0xc001bc3fe0 pc=0x4bf321 created by github.com/cockroachdb/pebble.(*DB).maybeScheduleCompactionPicker github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1641 +0x5c5 goroutine 1 [select]: github.com/cockroachdb/cockroach/pkg/cli.waitForShutdown(0xc000814f60, 0xc0011942d0, 0xc0003e6d20, 0xc001b30480, 0xc000425360) github.com/cockroachdb/cockroach/pkg/cli/start.go:727 +0x1c5 github.com/cockroachdb/cockroach/pkg/cli.runStart(0x91bf060, {0xc0011dfa10, 0x4ed0a81, 0x4}, 0x1) github.com/cockroachdb/cockroach/pkg/cli/start.go:674 +0x82a github.com/cockroachdb/cockroach/pkg/cli.runStartSingleNode(0xc001019a78, {0xc0005b24a0, 0x0, 0x2}) github.com/cockroachdb/cockroach/pkg/cli/start.go:339 +0x155 github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1(0xc001199300, {0xc0005b24a0, 0xc001019cc0, 0xc001019c98}) github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:67 +0x43 github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeShoutError.func1(0x91bf060, {0xc0005b24a0, 0x0, 0x2}) github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/shout.go:24 +0x22 github.com/spf13/cobra.(*Command).execute(0x91bf060, {0xc0005b2440, 0x2, 0x2}) github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856 +0x60e github.com/spf13/cobra.(*Command).ExecuteC(0x91bdee0) github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974 +0x3bc github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902 github.com/cockroachdb/cockroach/pkg/cli.Run(...) github.com/cockroachdb/cockroach/pkg/cli/cli.go:298 github.com/cockroachdb/cockroach/pkg/cli.doMain(0x91bf060, {0xc0009bc8aa, 0x11}) github.com/cockroachdb/cockroach/pkg/cli/cli.go:140 +0x1e9 github.com/cockroachdb/cockroach/pkg/cli.Main() github.com/cockroachdb/cockroach/pkg/cli/cli.go:65 +0x149 main.main() main/pkg/cmd/cockroach/main.go:26 +0x17 goroutine 6 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon() github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x5b created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x25 goroutine 7 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher() github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0xe5 created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x33 goroutine 34 [syscall]: os/signal.signal_recv() GOROOT/src/runtime/sigqueue.go:169 +0x98 os/signal.loop() GOROOT/src/os/signal/signal_unix.go:24 +0x19 created by os/signal.Notify.func1.1 GOROOT/src/os/signal/signal.go:151 +0x2c goroutine 66 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1() github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:165 +0x127 created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:157 +0x25 goroutine 24 [select]: github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1() github.com/cockroachdb/cockroach/pkg/cli/cli.go:330 +0x9d created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup github.com/cockroachdb/cockroach/pkg/cli/cli.go:328 +0x112 goroutine 15 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8ec00) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 16 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8ecc0) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 82 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8ed80) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 83 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f140) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 84 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f200) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 85 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f2c0) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 86 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f380) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 87 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f440) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 88 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f500) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 89 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f5c0) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 90 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f680) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 91 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f740) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 92 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f800) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 93 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f8c0) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 94 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8f980) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 95 [chan receive]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000d8fa40) github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187 goroutine 96 [select]: go.opencensus.io/stats/view.(*worker).start(0xc000d90400) go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:276 +0xb9 created by go.opencensus.io/stats/view.init.0 go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x92 goroutine 97 [select]: github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2() github.com/cockroachdb/cockroach/pkg/cli/cli.go:352 +0xab created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup github.com/cockroachdb/cockroach/pkg/cli/cli.go:350 +0x1f6 goroutine 98 [chan receive]: github.com/getsentry/sentry-go.(*HTTPTransport).worker(0xc000da3570) github.com/getsentry/sentry-go/external/com_github_getsentry_sentry_go/transport.go:387 +0x129 created by github.com/getsentry/sentry-go.(*HTTPTransport).Configure.func1 github.com/getsentry/sentry-go/external/com_github_getsentry_sentry_go/transport.go:255 +0x5f goroutine 114 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017de820, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:190 +0x725 goroutine 115 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df520, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 116 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df5f0, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 117 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df6c0, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 118 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df790, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 119 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df860, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 120 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017df930, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 121 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017dfa00, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 122 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017dfad0, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 123 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017dfba0, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 124 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017dfc70, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 125 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0017dfd40, {0x63c3dd0, 0xc000773ac0}) github.com/cockroachdb/cockroach/pkg/util/log/file_log_gc.go:25 +0x85 created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig github.com/cockroachdb/cockroach/pkg/util/log/flags.go:293 +0x1065 goroutine 126 [runnable]: syscall.Syscall6(0x106, 0xffffffffffffff9c, 0xc0004d0320, 0xc0008f46b8, 0x0, 0x0, 0x0) GOROOT/src/syscall/asm_linux_amd64.s:43 +0x5 syscall.fstatat(0xc00057b168, {0xc00000c9d8, 0xc00057b188}, 0xc0008f46b8, 0xc0008f4000) GOROOT/src/syscall/zsyscall_linux_amd64.go:1441 +0x10f syscall.Stat(...) GOROOT/src/syscall/syscall_linux_amd64.go:66 os.statNolog.func1(...) GOROOT/src/os/stat_unix.go:33 os.ignoringEINTR(...) GOROOT/src/os/file_posix.go:246 os.statNolog({0xc00000c9d8, 0x13}) GOROOT/src/os/stat_unix.go:32 +0x58 os.Stat({0xc00000c9d8, 0x18}) GOROOT/src/os/stat.go:13 +0x34 github.com/shirou/gopsutil/v3/internal/common.PathExists(...) github.com/shirou/gopsutil/v3/internal/common/external/com_github_shirou_gopsutil_v3/internal/common/common.go:308 github.com/shirou/gopsutil/v3/disk.LabelWithContext({0xc00057b4a0, 0xc00057b418}, {0xc001b185cd, 0xffffffffffffffff}) github.com/shirou/gopsutil/v3/disk/external/com_github_shirou_gopsutil_v3/disk/disk_linux.go:482 +0xb5 github.com/shirou/gopsutil/v3/disk.IOCountersWithContext({0x63c3e78, 0xc00112fbf0}, {0x0, 0x10, 0x0}) github.com/shirou/gopsutil/v3/disk/external/com_github_shirou_gopsutil_v3/disk/disk_linux.go:439 +0x61f github.com/cockroachdb/cockroach/pkg/server/status.getDiskCounters({0x63c3e78, 0xc00112fbf0}) github.com/cockroachdb/cockroach/pkg/server/status/disk_counters.go:24 +0x45 github.com/cockroachdb/cockroach/pkg/server/status.getSummedDiskCounters({0x63c3e78, 0xc00112fbf0}) github.com/cockroachdb/cockroach/pkg/server/status/runtime.go:627 +0x68 github.com/cockroachdb/cockroach/pkg/server/status.NewRuntimeStatSampler({0x63c3e78, 0xc00112fbf0}, 0xc001b2ac80) github.com/cockroachdb/cockroach/pkg/server/status/runtime.go:341 +0x48e github.com/cockroachdb/cockroach/pkg/server.NewServer({{0xc000c98a80, 0xc0009428c0, 0xc00002c5a0, 0xc0008fc4b0, 0xc000b85590, 0xc000b85578, {0xc00002c5a0, {0x6319da0, 0xc0008fc4b0}, 0x0, ...}, ...}, ...}, ...) github.com/cockroachdb/cockroach/pkg/server/server.go:301 +0xa95 github.com/cockroachdb/cockroach/pkg/cli.runStart.func3.2(0xc0011942d0, 0xc0003c86a0, 0xc000425360, {0x63c3e78, 0xc00072a0f0}, 0x1, {0x243a811e, 0xedb0f4905, 0x0}) github.com/cockroachdb/cockroach/pkg/cli/start.go:613 +0x95 github.com/cockroachdb/cockroach/pkg/cli.runStart.func3() github.com/cockroachdb/cockroach/pkg/cli/start.go:669 +0xf6 created by github.com/cockroachdb/cockroach/pkg/cli.runStart github.com/cockroachdb/cockroach/pkg/cli/start.go:584 +0x7c5 goroutine 129 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b545a0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0xc000c807d0}}, 0xc000c807a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x51c6358) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 130 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54540}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0x0}}, 0xc000c80fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 131 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54660}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0x0}}, 0xc000c817a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 132 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54600}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0x0}}, 0xc000c81fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 133 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b546c0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x0, 0x0}}, 0xc000c827a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 134 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b548a0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x96ba46947e5d61d2, 0x57953d64295b3870}}, 0xc001072fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xdd50d8dd592b9cee) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 135 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc00046a090}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0xfc7beb4081cc0fc4, 0xc38771d9361f27ab}}, 0xc0010737a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x30d9842a59a1429a) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 136 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54780}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x4c5b3aaaa7577491, 0x50ccf64468dd0d02}}, 0xc001073fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc1a8de6e826d5bd1) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 137 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b547e0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0xeed38091b522ea1a, 0x7a47002f57ad8215}}, 0xc0010747a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xce4c662ff0addf16) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 138 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54840}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x11e374a594398d4c, 0xb957f875a156374a}}, 0xc001074fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x1002e5a3306b74cb) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 139 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54720}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x2b2ddca3f0e580a1, 0x53ad38379bbf396f}}, 0xc0010757a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xb8b10cde283db44e) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 140 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54900}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0xbe78e69c5c420883, 0x8dfe625f833c720a}}, 0xc001075fa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x540159129e043a73) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 141 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc00046a0f0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x693e9ffcfad0c73d, 0xafea905e8fe2fe7d}}, 0xc0010767a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x5e2bfaf1b27ee713) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 142 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b54960}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x3a320e63ff44129e, 0x8e056fe2ef839f42}}, 0xc00128cfa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xcd9b6896cc3e6407) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 143 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc00046a150}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x567007209e2f78c3, 0x9cf21c7f7c8c5d89}}, 0xc00128d7a8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x32bc803ca25b3d9f) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 144 [chan receive]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x63c3e78, 0xc001b549c0}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:300 +0x5b runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c7a0, 0x2703beac0ab70697, 0x5ae547673a6c7f9c}}, 0xc00128dfa8) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x162bbcdd601aceb5) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:299 +0x58 created by github.com/cockroachdb/pebble.(*tableCacheShard).init github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:291 +0xd1 goroutine 146 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:302 +0xe9 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:297 +0x85 goroutine 147 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:72 +0xe8 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:67 +0x65 goroutine 148 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:72 +0xe8 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:67 +0x65 goroutine 149 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:72 +0xe8 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:67 +0x65 goroutine 52 [sync.Cond.Wait]: sync.runtime_notifyListWait(0xc0004aa0e0, 0x0) GOROOT/src/runtime/sema.go:513 +0x13d sync.(*Cond).Wait(0x4770aa0) GOROOT/src/sync/cond.go:56 +0x8c github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:193 github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0004aa000, {0x47e52a0, 0x9bf0860}) github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:396 +0x479 runtime/pprof.Do({0x63c3e08, 0xc000074120}, {{0xc00046c680, 0xc000c18000, 0x0}}, 0xc000781fc0) GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 github.com/cockroachdb/pebble/record.NewLogWriter.func2() github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:314 +0x5c created by github.com/cockroachdb/pebble/record.NewLogWriter github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:313 +0x329 goroutine 103 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:72 +0xe8 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:67 +0x65 goroutine 151 [syscall]: syscall.Syscall(0x107, 0xffffffffffffff9c, 0xc000b34990, 0x0) GOROOT/src/syscall/asm_linux_amd64.s:20 +0x5 syscall.unlinkat(0xc000da0cd8, {0xc000b34960, 0x7fff94bca090}, 0xffffffffffffffff) GOROOT/src/syscall/zsyscall_linux_amd64.go:127 +0xd4 syscall.Unlink(...) GOROOT/src/syscall/syscall_linux.go:185 os.Remove.func1(...) GOROOT/src/os/file_unix.go:302 os.ignoringEINTR(...) GOROOT/src/os/file_posix.go:246 os.Remove({0xc000b34960, 0x48574e}) GOROOT/src/os/file_unix.go:301 +0x45 github.com/cockroachdb/pebble/vfs.defaultFS.Remove({}, {0xc000b34960, 0x7f6ae1fc0108}) github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/vfs.go:184 +0x25 github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Remove.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:445 +0x37 github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).timeFilesystemOp(0x2f6700c0000c3d40, {0xc000b34960, 0x1}, 0xc0000c3d48) github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:286 +0x46 github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Remove(0x0, {0xc000b34960, 0x0}) github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:444 +0x69 github.com/cockroachdb/pebble/vfs.(*enospcFS).Remove(0xc000baa000, {0xc000b34960, 0x29}) github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:247 +0x4f github.com/cockroachdb/pebble/internal/base.DeleteCleaner.Clean(...) github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/cleaner.go:25 github.com/cockroachdb/pebble.(*DB).deleteObsoleteFile(0xc0001fe480, 0x0, 0x1, {0xc000b34960, 0x29}, 0x2) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2796 +0x96 github.com/cockroachdb/pebble.(*DB).paceAndDeleteObsoleteFiles(0xc0001fe480, 0x0, {0xc0006be1e0, 0x3, 0xc001070798}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2764 +0x387 created by github.com/cockroachdb/pebble.(*DB).doDeleteObsoleteFiles github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2739 +0x865 goroutine 105 [select]: github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:72 +0xe8 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:67 +0x65 goroutine 154 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func2({0xc000354f00, 0x77beff1ab3329d72}) github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:541 +0x45 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x16f created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:483 +0x445 goroutine 155 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc000355500, 0xc000c86f20}) github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/infostore.go:192 +0x8a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x16f created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:483 +0x445 ```

Here's the highlight:

fatal error: Go pointer stored into non-Go memory

goroutine 153 [running]:
runtime.throw({0x4f75f26, 0xc001bc6188})
    GOROOT/src/runtime/panic.go:1198 +0x71 fp=0xc001bc12f8 sp=0xc001bc12c8 pc=0x48ac91
runtime.cgoCheckBits(0x4c166a, 0xc0010ca9c0, 0xc000089800, 0xc001bc13e0)
    GOROOT/src/runtime/cgocheck.go:192 +0xc5 fp=0xc001bc1348 sp=0xc001bc12f8 pc=0x458b85
runtime.cgoCheckTypedBlock(0x7f6ab2614000, 0x51bf6e8, 0xc001bc1410, 0x4bf487)
    GOROOT/src/runtime/cgocheck.go:118 +0x57 fp=0xc001bc13d0 sp=0xc001bc1348 pc=0x4587b7
runtime.cgoCheckMemmove(0x4d54760, 0x4889760, 0xc000673020, 0x7f6ab2614000, 0xc000673020)
    GOROOT/src/runtime/cgocheck.go:76 +0x6c fp=0xc001bc1420 sp=0xc001bc13d0 pc=0x45866c
runtime.typedmemmove(0x4d54760, 0x7f6ab2614000, 0xc001bc1490)
    GOROOT/src/runtime/mbarrier.go:173 +0x87 fp=0xc001bc1458 sp=0xc001bc1420 pc=0x4677e7
github.com/cockroachdb/pebble/internal/cache.newEntry(0xc0012cb450, {{0x7f6ab2613000, 0xc001bc15e0}, 0x11a4afd}, 0x53)
    github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/entry.go:68 +0x108 fp=0xc001bc1500 sp=0xc001bc1458 pc=0x1186aa8

Environment:

$ cockroach version
Build Tag:        v22.1.9
Build Time:       2022/10/17 18:24:57
Distribution:     CCL
Platform:         linux amd64 (x86_64-pc-linux-gnu)
Go Version:       go1.17.11
C Compiler:       gcc 6.5.0
Build Commit ID:  e438c2f89282e607e0e6ca1d38b2e0a622f94493
Build Type:       release

See above for AMI info -- it's Ubuntu 22.04 LTS.

Additional context

Caveat: I'm not very experienced with Go myself. I gather that this failure means Go thinks this code is violating the rules for passing pointers. I see that the Pebble code is aware of the constraints here and it's believed safe because "the Cache guarantees that there are other pointers to the shard which will keep it alive". That seems to imply that the only risk here is that references to Go objects in C memory would be missed by the GC and potentially freed while still in use. But the rules also say:

Note: the current implementation has a bug. While Go code is permitted to write nil or a C pointer (but not a Go pointer) to C memory, the current implementation may sometimes cause a runtime error if the contents of the C memory appear to be a Go pointer. Therefore, avoid passing uninitialized C memory to Go code if the Go code is going to store pointer values in it. Zero out the memory in C before passing it to Go.

There's some more information in this thread and this issue. These threads make me wonder what might happen if you're running in a configuration with non-Go-managed memory for the cache, and the memory is uninitialized, and it happens to contain Go pointers. Might the Go GC incorrectly follow these pointers? Could that not cause all kinds of problems, including cases where GC finds an apparently-referenced object that's actually free? (context: I'm seeing a bunch of crashes from the Go runtime memory allocator that say exactly that.)

Jira issue: CRDB-21719

blathers-crl[bot] commented 1 year ago

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

@cockroachdb/replication (found keywords: raft)
@cockroachdb/storage (found keywords: pebble,compaction)
@nicktrav (commented on cockroachdb/cockroach#82958)
@knz (commented on cockroachdb/cockroach#82958)

If we have not gotten back to your issue within a few business days, you can try the following:

Join our community slack channel and ask on #cockroachdb.
Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/replication

knz commented 1 year ago

That last part of your comment, about what the go GC might or might not think of uninitialized memory we allocate via cgo, looks like a technical problem where @petermattis would be the best expert.

davepacheco commented 1 year ago

I dug a little further and found that the manual allocation path appears to work around this problem: https://github.com/cockroachdb/pebble/blob/936e011bb911a82b8711196b7c042c7d57b7c59f/internal/manual/manual.go#L28-L40

I'm still not sure the Pebble code is really safe? The Go guidelines don't say that the only way this can fail is if there's no other reference to the object then it might get collected. It just says "Go code may not store a Go pointer in C memory...programs that break these rules are likely to fail in unexpected and unpredictable ways."

petermattis commented 1 year ago

These threads make me wonder what might happen if you're running in a configuration with non-Go-managed memory for the cache, and the memory is uninitialized, and it happens to contain Go pointers. Might the Go GC incorrectly follow these pointers? Could that not cause all kinds of problems, including cases where GC finds an apparently-referenced object that's actually free? (context: I'm seeing a bunch of crashes from the Go runtime memory allocator that say exactly that.)

I'm pretty sure that the Go GC never steps through non-Go memory. That is, it only traces memory references through the Go heap, Go globals, and Go stacks. It is true that Pebble violates the cgo rules about storing a Go pointer in C allocated memory. This is discussed at https://github.com/cockroachdb/pebble/blob/936e011bb911a82b8711196b7c042c7d57b7c59f/internal/cache/entry.go#L27-L41:

// entry holds the metadata for a cache entry. The memory for an entry is
// allocated from manually managed memory.
//
// Using manual memory management for entries is technically a volation of the
// Cgo pointer rules:
//
//   https://golang.org/cmd/cgo/#hdr-Passing_pointers
//
// Specifically, Go pointers should not be stored in C allocated memory. The
// reason for this rule is that the Go GC will not look at C allocated memory
// to find pointers to Go objects. If the only reference to a Go object is
// stored in C allocated memory, the object will be reclaimed. The shard field
// of the entry struct points to a Go allocated object, thus the
// violation. What makes this "safe" is that the Cache guarantees that there
// are other pointers to the shard which will keep it alive.

Due to the violation of the Cgo pointer rules, running cockroach with GODEBUG=cgocheck=2 isn't going to work. That said, we've never seen evidence that this violation of the Cgo pointer rules actually causes problems when you're not running with cgocheck=2. As the comment explains, this violation appears to be copacetic as long as Go does not have a moving GC.

srosenberg commented 1 year ago

As Peter already explained, the side-effect of enabling cgocheck=2 is a write barrier which crashes the program whenever a Go pointer is attempted to be written into non-Go memory [1].

[1] https://github.com/golang/go/blob/c8057d85692c01e96d3c04815a0a364c7cfb4d90/src/runtime/cgocheck.go#L55

davepacheco commented 1 year ago

Right. It's an unfortunate side effect of this behavior that we cannot use the cgocheck=2 diagnostic to debug other issues like #82958 because of the false positives from this code path. But it doesn't seem like there's anything we can do about that. Thanks for the update.

tbg commented 1 year ago

@davepacheco I don't know if it's worth your while, but it should be possible to patch pebble to avoid allocating manually. Basically changing Free to be a no-op and making New just allocate using Go around here. I'm not sure if that might have other unintended side effects, certainly wouldn't do it in Prod but it sounded like you were mostly using CRDB for running test suites so at least for figuring out whether cgocheck passes this could be worth a try.

petermattis commented 1 year ago

@tbg Pebble already does that if you disable cgo (e.g. via setting CGO_ENABLED=0 during compilation). @davepacheco If you want to see how that is done, grep for cgo in the internal/{cache,manual,rawalloc} directories. Note that performance is a lot worse in this mode, but it may be adequate for small testing usage.

cockroachdb / cockroach

crash with GODEBUG=cgocheck=2: Go pointer stored into non-Go memory #92361