search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.15k stars 3.81k forks source link

demo: seg fault at interactive prompt without specific input #62283

Closed rytaft closed 1 year ago

rytaft commented 3 years ago

Here is the output of my terminal, using the latest master:

Rebeccas-MacBook-Pro:cockroach becca$ ./cockroach demo --nodes 9 --global
#
# Welcome to the CockroachDB demo database!
#
# You are connected to a temporary, in-memory CockroachDB cluster of 9 nodes.
#
# This demo session will attempt to enable enterprise features
# by acquiring a temporary license from Cockroach Labs in the background.
# To disable this behavior, set the environment variable
# COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING=true.
#
# Beginning initialization of the movr dataset, please wait...
#
# The cluster has been preloaded with the "movr" dataset
# (MovR is a fictional vehicle sharing company).
#
# Reminder: your changes to data stored in the demo session will not be saved!
#
# Connection parameters:
#   (console) http://127.0.0.1:8080/demologin?password=demo49832&username=demo
#   (sql)     postgres://demo:demo49832@?host=%2Fvar%2Ffolders%2Fwc%2Fhvy8pyy128z3v_f9lhvxw99h0000gn%2FT%2Fdemo943322695&port=26257
#   (sql/tcp) postgres://demo:demo49832@127.0.0.1:26257?sslmode=require
# 
# To display connection parameters for other nodes, use \demo ls.
#
# The user "demo" with password "demo49832" has been created. Use it to access the Web UI!
#
# Server version: CockroachDB CCL v21.1.0-alpha.3-1597-g24e76d3990 (x86_64-apple-darwin20.3.0, built , go1.15.5) (same version as client)
# Cluster ID: 73bb9025-3ed7-4f98-90ea-584cc29fa13a
# Organization: Cockroach Demo
#
# Enter \? for a brief introduction.
#
demo@127.0.0.1:26257/movr> SHOW REGIONS;
     region    |  zones  | database_names | primary_region_of
---------------+---------+----------------+--------------------
  europe-west1 | {b,c,d} | {}             | {}
  us-east1     | {b,d,c} | {}             | {}
  us-west1     | {b,a,c} | {}             | {}
(3 rows)

Time: 1.198s total (execution 1.198s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW TABLES;
  schema_name |         table_name         | type  | owner | estimated_row_count | locality
--------------+----------------------------+-------+-------+---------------------+-----------
  public      | promo_codes                | table | demo  |                1000 | NULL
  public      | rides                      | table | demo  |                 500 | NULL
  public      | user_promo_codes           | table | demo  |                   0 | NULL
  public      | users                      | table | demo  |                  50 | NULL
  public      | vehicle_location_histories | table | demo  |                1000 | NULL
  public      | vehicles                   | table | demo  |                  15 | NULL
(6 rows)

Time: 420ms total (execution 365ms / network 55ms)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION “us-east1”
;
invalid syntax: statement ignored: at or near "-": syntax error
SQLSTATE: 42601
DETAIL: source SQL:
ALTER DATABASE movr PRIMARY REGION “us-east1”
                                        ^
demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION "us-east1";
ALTER DATABASE PRIMARY REGION

Time: 6.719s total (execution 3.213s / network 3.507s)

demo@127.0.0.1:26257/movr> SHOW DATABASES;
  database_name | owner | primary_region |  regions   | survival_goal
----------------+-------+----------------+------------+----------------
  defaultdb     | root  | NULL           | {}         | NULL
  movr          | demo  | us-east1       | {us-east1} | zone
  postgres      | root  | NULL           | {}         | NULL
  system        | node  | NULL           | {}         | NULL
(4 rows)

Time: 1.388s total (execution 1.388s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW REGIONS;
     region    |  zones  | database_names | primary_region_of
---------------+---------+----------------+--------------------
  europe-west1 | {b,c,d} | {}             | {}
  us-east1     | {b,d,c} | {movr}         | {movr}
  us-west1     | {b,a,c} | {}             | {}
(3 rows)

Time: 2.236s total (execution 2.235s / network 0.001s)

demo@127.0.0.1:26257/movr> SHOW TABLES;
  schema_name |         table_name         | type  | owner | estimated_row_count |              locality
--------------+----------------------------+-------+-------+---------------------+--------------------------------------
  public      | promo_codes                | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | rides                      | table | demo  |                 500 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | user_promo_codes           | table | demo  |                   0 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | users                      | table | demo  |                  50 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicle_location_histories | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicles                   | table | demo  |                  15 | REGIONAL BY TABLE IN PRIMARY REGION
(6 rows)

Time: 1.015s total (execution 1.015s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW ZONE CONFIGURATION FOR DATABASE movr;
     target     |                 raw_config_sql
----------------+-------------------------------------------------
  DATABASE movr | ALTER DATABASE movr CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 90000,
                |     num_replicas = 3,
                |     num_voters = 3,
                |     constraints = '{+region=us-east1: 1}',
                |     voter_constraints = '[+region=us-east1]',
                |     lease_preferences = '[[+region=us-east1]]'
(1 row)

Time: 345ms total (execution 344ms / network 1ms)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION "europe-west1";
ERROR: region "europe-west1" has not been added to the database
SQLSTATE: 42602
HINT: you must add the region to the database before setting it as primary region, using ALTER DATABASE movr ADD REGION "europe-west1"
demo@127.0.0.1:26257/movr> ALTER DATABASE movr add REGION "europe-west1";
ALTER DATABASE ADD REGION

Time: 9.973s total (execution 1.830s / network 8.143s)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr add REGION "us-west1";
ALTER DATABASE ADD REGION

Time: 9.641s total (execution 1.804s / network 7.836s)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr SURVIVE REGION FAILURE;
ALTER DATABASE SURVIVE

Time: 9.183s total (execution 5.443s / network 3.741s)

demo@127.0.0.1:26257/movr> alter table rides set locality regional by row;
NOTICE: LOCALITY changes will be finalized asynchronously; further schema changes on this table may be restricted until the job completes
ALTER TABLE SET LOCALITY

Time: 30.254s total (execution 3.117s / network 27.136s)

demo@127.0.0.1:26257/movr> show tables;
  schema_name |         table_name         | type  | owner | estimated_row_count |              locality
--------------+----------------------------+-------+-------+---------------------+--------------------------------------
  public      | promo_codes                | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | rides                      | table | demo  |                 500 | REGIONAL BY ROW
  public      | user_promo_codes           | table | demo  |                   0 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | users                      | table | demo  |                  50 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicle_location_histories | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicles                   | table | demo  |                  15 | REGIONAL BY TABLE IN PRIMARY REGION
(6 rows)

Time: 1.129s total (execution 1.128s / network 0.000s)

demo@127.0.0.1:26257/movr> unexpected fault address 0xfffffff800000019
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xfffffff800000019 pc=0x5658770]

goroutine 2093790 [running]:
runtime.throw(0x865aaf3, 0x5)
    /usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc0338bdd18 sp=0xc0338bdce8 pc=0x403c7b2
runtime.sigpanic()
    /usr/local/opt/go/libexec/src/runtime/signal_unix.go:749 +0x3e5 fp=0xc0338bdd48 sp=0xc0338bdd18 pc=0x40532e5
github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).PublicColumns(0xc0005adb00, 0x0, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/table_desc.go:306 +0x30 fp=0xc0338bdd68 sp=0xc0338bdd48 pc=0x5658770
github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys.MakeNameMetadataKey(0xc000d04e40, 0xc000d04e40, 0x0, 0x1d00000001, 0xc042661590, 0x4, 0x0, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys/keys.go:284 +0x1d6 fp=0xc0338bde20 sp=0xc0338bdd68 pc=0x56c5d16
github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkv.(*oneLevelUncachedDescGetter).GetNamespaceEntries(0xc03ea2a520, 0x9495780, 0xc03205b8c0, 0xc03ed993c0, 0x1, 0x1, 0x3, 0x0, 0xc0325ddae0, 0xc02d8381a0, ...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkv/desc_getter.go:129 +0x91e fp=0xc0338be008 sp=0xc0338bde20 pc=0x56d2e3e
github.com/cockroachdb/cockroach/pkg/sql/catalog.(*validationDescGetterImpl).addNamespaceEntries(0xc03ea2a580, 0x9495780, 0xc03205b8c0, 0xc014fbb8b0, 0x1, 0x1, 0x16a530d8, 0xc03ea2a520, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/validate.go:293 +0x472 fp=0xc0338be108 sp=0xc0338be008 pc=0x55a6ef2
github.com/cockroachdb/cockroach/pkg/sql/catalog.Validate(0x9495780, 0xc03205b8c0, 0x16a530d8, 0xc03ea2a520, 0xc00000001f, 0xc014fbb8b0, 0x1, 0x1, 0xc024abe8b0, 0x24abe8b004076f20)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/validate.go:81 +0x397 fp=0xc0338be1f0 sp=0xc0338be108 pc=0x55a5b77
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).ValidateUncommittedDescriptors(0xc030cb9570, 0x9495780, 0xc03205b8c0, 0xc048bc3e60, 0x24abe8b0, 0xed7e72495)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:1531 +0x1ea fp=0xc0338be278 sp=0xc0338be1f0 pc=0x573794a
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc032182d80, 0xc031a70328)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:755 +0x69 fp=0xc0338be2c0 sp=0xc0338be278 pc=0x70080a9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransaction(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc031a703e0, 0x4013290, 0xc0338be380, 0x570018d)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:738 +0x8c fp=0xc0338be318 sp=0xc0338be2c0 pc=0x7007f2c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).handleAutoCommit(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc0338be420, 0x40, 0xc00bd20900, 0x8d50c08)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1443 +0xba fp=0xc0338be3b0 sp=0xc0338be318 pc=0x700cfda
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState.func7(0xc030cb9300, 0xc0338bf270, 0xc0338bf260, 0xc0338bf1f0, 0xc0338be8d8, 0xc0338bf250, 0x94466c0, 0xb3fc8db)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:444 +0xc9 fp=0xc0338be420 sp=0xc0338be3b0 pc=0x71ba809
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0x87e4979, 0x74, 0x4, 0x1, 0xc0317dc0f0, ...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:702 +0x124f fp=0xc0338bf1e8 sp=0xc0338be420 pc=0x70058ef
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0x94b4040, 0xc0199f12c0, 0x87e4979, 0x74, 0x4, 0x1, 0xc0317dc0f0, ...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:120 +0xb14 fp=0xc0338bf718 sp=0xc0338bf1e8 pc=0x7003f94
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0xc0317dc0f0, 0xc031fe5b80, 0x4, 0x4, 0x0, 0x0, 0x0, ...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:207 +0x14e fp=0xc0338bf850 sp=0xc0338bf718 pc=0x70043ee
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2(0x0, 0x0, 0x0, 0x14d22b70, 0xed7e72495, 0x0, 0xc030cb9300, 0xc0338bfc18, 0xc0338bfc08, 0x2, ...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1583 +0x47b fp=0xc0338bfa68 sp=0xc0338bf850 pc=0x71b8efb
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1585 +0x45c fp=0xc0338bfe90 sp=0xc0338bfa68 pc=0x6ff853c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc030cb9300, 0x9495780, 0xc031cb59e0, 0xc001713180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1437 +0x228 fp=0xc0338bff30 sp=0xc0338bfe90 pc=0x6ff7e68
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc030cb9300, 0x9495780, 0xc031cb59e0, 0xc000469d40, 0xc03e0b05c0, 0xc031cab740, 0x0, 0xc041bee3c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:202 +0x65 fp=0xc0338bffa0 sp=0xc0338bff30 pc=0x71c6305
runtime.goexit()
    /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0338bffa8 sp=0xc0338bffa0 pc=0x4075761
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:201 +0x2dc

goroutine 1 [syscall]:
github.com/knz/go-libedit/unix._C2func_go_libedit_gets(0x37fe0000, 0x3cb77000, 0x0, 0x0, 0xc027587740, 0xc027587744, 0xc000000001, 0x0, 0x0, 0x0)
    _cgo_gotypes.go:190 +0x56
github.com/knz/go-libedit/unix.EditLine.GetLine.func1(0xc00e0e0e00, 0xc027587740, 0xc027587744, 0xc00e0e0e00, 0xc006cd16a0, 0x78d2c33)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:281 +0xe7
github.com/knz/go-libedit/unix.EditLine.GetLine(0x0, 0xb812ea0, 0x0, 0x0, 0xb812ea0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:281 +0xaa
github.com/cockroachdb/cockroach/pkg/cli.(*cliState).doReadLine(0xc00b83f770, 0x6, 0x5)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:953 +0x5e5
github.com/cockroachdb/cockroach/pkg/cli.(*cliState).doRunShell(0xc00b83f770, 0x0, 0xc000010010, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1482 +0xb4
github.com/cockroachdb/cockroach/pkg/cli.runInteractive(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1450
github.com/cockroachdb/cockroach/pkg/cli.runClient(0xb77e860, 0xc00b7ef720, 0xc000010010, 0x52, 0xc00b3ef968)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1715 +0xea
github.com/cockroachdb/cockroach/pkg/cli.runDemo(0xb77e860, 0x9440bc0, 0xc000d82360, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/demo.go:367 +0x815
github.com/cockroachdb/cockroach/pkg/cli.init.4.func1(0xb77e860, 0xc000591f80, 0x0, 0x3, 0x1, 0x405436c)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/demo.go:55 +0x33
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0xb77e860, 0xc000591f80, 0x0, 0x3, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:194 +0x79
github.com/spf13/cobra.(*Command).execute(0xb77e860, 0xc000591f20, 0x3, 0x3, 0xb77e860, 0xc000591f20)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xb77cf60, 0xc0000740f0, 0xc000447e9a, 0x4)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:312
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xb77e860, 0xc000447e9a, 0x4, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:112 +0x11a
github.com/cockroachdb/cockroach/pkg/cli.Main()
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:60 +0x13a
main.main()
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26 +0x25

goroutine 31 [chan receive, 17 minutes]:
github.com/cockroachdb/cmux.muxListener.Accept(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/pkg/rpc.delayingListener.Accept(0x946fac0, 0xc0026353e0, 0xc0040e7df0, 0x18, 0xc0002a5080, 0x483af9e)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:858 +0x3b
google.golang.org/grpc.(*Server).Serve(0xc0041129c0, 0x94701c0, 0xc003377140, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:621 +0x216
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x9495780, 0xc001234ed0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1972 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000e8b980, 0x9495780, 0xc001234ed0, 0x0, 0xc002695660)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 8 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x73
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35

goroutine 9 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d

goroutine 34 [syscall, 17 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/opt/go/libexec/src/runtime/sigqueue.go:144 +0x9d
os/signal.loop()
    /usr/local/opt/go/libexec/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
    /usr/local/opt/go/libexec/src/os/signal/signal.go:150 +0x45

goroutine 52 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0009f2320)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 16 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c180)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c180, 0xc0003a3c60, 0x1, 0x1, 0xc0005a97b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787100)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 15 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88060)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88060, 0xc0003a3c60, 0x1, 0x1, 0xc0005a8fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001786200)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 370 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e086f0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 44 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func1(0x9495780, 0xc0013407e0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1310 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001468800, 0x9495780, 0xc0013407e0, 0x0, 0xc002662b80)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 66 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c880c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c880c0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a87b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787200)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 67 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88120)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88120, 0xc0003a3c60, 0x1, 0x1, 0xc000088fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787300)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 1749345 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/opt/go/libexec/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc002b86fe0, 0x3f4)
    /usr/local/opt/go/libexec/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc002b86fd0)
    /usr/local/opt/go/libexec/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/pebble/internal/record.(*flusherCond).Wait(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:193
github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop(0xc002b86f00, 0x9495780, 0xc0143abe00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:396 +0x10b
runtime/pprof.Do(0x9495780, 0xc0143abe00, 0xc0003a32c0, 0x1, 0x1, 0xc0097227b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2(0xc002b86f00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:314 +0x89
created by github.com/cockroachdb/pebble/internal/record.NewLogWriter
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:313 +0x2ea

goroutine 43 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0xe9b6ed8, 0x72, 0x0)
    /usr/local/opt/go/libexec/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc001468718, 0x72, 0x0, 0x0, 0x865d60a)
    /usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc001468700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/opt/go/libexec/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc001468700, 0x37e11d600, 0x0, 0x0)
    /usr/local/opt/go/libexec/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc001333e40, 0xc004152e10, 0xc004152e18, 0x28)
    /usr/local/opt/go/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc001333e40, 0x8d4eec8, 0xc00128a200, 0x94fcac0, 0xc037429fa0)
    /usr/local/opt/go/libexec/src/net/tcpsock.go:261 +0x65
github.com/cockroachdb/cmux.(*cMux).Serve(0xc00128a200, 0x0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.2.1()
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1977 +0x2f
sync.(*Once).doSlow(0xc0009a1900, 0xc000ad4f20)
    /usr/local/opt/go/libexec/src/sync/once.go:66 +0xec
sync.(*Once).Do(...)
    /usr/local/opt/go/libexec/src/sync/once.go:57
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.2(0x9495780, 0xc001b90960)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1976 +0x74
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019bc500, 0x9495780, 0xc001b90960, 0x0, 0xc002662b60)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 70 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x94956c0, 0xc00053e480)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:402 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x94956c0, 0xc00053e480, 0x0, 0xc0006765e0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 71 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc000dd0100, 0x9495700, 0xc0000740f0, 0xc000cfa0f0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:235 +0x159
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:232 +0x65

goroutine 72 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x9495700, 0xc0000740f0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x17b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495700, 0xc0000740f0, 0x0, 0xc0009eefc0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 65 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e086a0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 64 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e08690)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 75 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000e97aa0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000e97aa0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a2fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787700)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 76 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c881e0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c881e0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a37b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787800)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 77 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88240)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88240, 0xc0003a3c60, 0x1, 0x1, 0xc0005a4fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787900)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 78 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c450)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c450, 0xc0003a3c60, 0x1, 0x1, 0xc0005a57b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787c00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 79 [sync.Cond.Wait, 17 minutes]:
runtime.goparkunlock(...)
    /usr/local/opt/go/libexec/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc00032afe0, 0x0)
    /usr/local/opt/go/libexec/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc00032afd0)
    /usr/local/opt/go/libexec/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/pebble/internal/record.(*flusherCond).Wait(...)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:193
github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop(0xc00032af00, 0x9495780, 0xc000d8c630)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:396 +0x10b
runtime/pprof.Do(0x9495780, 0xc000d8c630, 0xc0003a32c0, 0x1, 0x1, 0xc0005a5fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2(0xc00032af00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:314 +0x89
created by github.com/cockroachdb/pebble/internal/record.NewLogWriter
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:313 +0x2ea

goroutine 30 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func5(0x9495780, 0xc001234ed0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1351 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000e8b980, 0x9495780, 0xc001234ed0, 0x0, 0xc002694e20)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 81 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9990)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 82 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99a0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 83 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99b0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 84 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 85 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99d0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 86 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99e0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 87 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99f0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 88 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 89 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a10)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 90 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a20)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 91 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a30)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 92 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a40)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 93 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a50)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 94 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a60)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 95 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a70)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 96 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a80)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 3747 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc006468700)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1339 +0x193
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:295 +0x16fc

goroutine 98 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x8d58d50, 0xc000683d10, 0xc0003ee2a0, 0x9495780, 0xc000d183c0, 0x0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:230 +0x12d
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:223 +0xd0

goroutine 99 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x9495700, 0xc0000740f8)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:113 +0x65
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495700, 0xc0000740f8, 0x0, 0xc000d385a0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 100 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2001 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38600)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 101 [IO wait, 17 minutes]:
internal/poll.runtime_pollWait(0xe9b7530, 0x72, 0x0)
    /usr/local/opt/go/libexec/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000450798, 0x72, 0x0, 0x0, 0x865d60a)
    /usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000450780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/opt/go/libexec/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000450780, 0xc0014605a0, 0x4013bf8, 0xc000500000)
    /usr/local/opt/go/libexec/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000d385e0, 0xc000a115c0, 0x4013bf8, 0x30)
    /usr/local/opt/go/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000d385e0, 0x837db00, 0xc0014605a0, 0x7f46660, 0xb73dea0)
    /usr/local/opt/go/libexec/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc00144c000, 0x946dd80, 0xc000d385e0, 0x0, 0x0)
    /usr/local/opt/go/libexec/src/net/http/server.go:2937 +0x266
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2049 +0x3f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38640)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 102 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1902 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38b00)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 103 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x9495780, 0xc000d183c0)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1946 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38b60)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 104 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc001460600)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc001460600, 0xc0003a3c60, 0x1, 0x1, 0xc0005a7fb8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412200)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 105 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c450)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c450, 0xc0003a3c60, 0x1, 0x1, 0xc0005a77b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412300)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 106 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c780)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c780, 0xc0003a3c60, 0x1, 0x1, 0xc000a107b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412400)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 107 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8d140)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8d140, 0xc0003a3c60, 0x1, 0x1, 0xc000a177b8)
    /usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412500)
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
    /Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 0 [idle]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x130 pc=0x40655b0]

runtime stack:
runtime.throw(0x8708d39, 0x2a)
    /usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
    /usr/local/opt/go/libexec/src/runtime/signal_unix.go:726 +0x48c
runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc0005abb00, 0x0)
    /usr/local/opt/go/libexec/src/runtime/traceback.go:712 +0x50
runtime.traceback(...)
    /usr/local/opt/go/libexec/src/runtime/traceback.go:690
runtime.tracebackothers(0xc00bd20900)
    /usr/local/opt/go/libexec/src/runtime/traceback.go:944 +0x165
runtime.dopanic_m(0xc00bd20900, 0x403c7b2, 0xc0338bdce8, 0x865aaf3)
    /usr/local/opt/go/libexec/src/runtime/panic.go:1322 +0x2bf
runtime.fatalthrow.func1()
    /usr/local/opt/go/libexec/src/runtime/panic.go:1171 +0x5f
runtime.systemstack(0xec17740)
    /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
    /usr/local/opt/go/libexec/src/runtime/proc.go:1116

Jira issue: CRDB-6374

rytaft commented 3 years ago

cc @postamar @ajwerner not sure if this actually has anything to do with schema stuff, but it appears that the seg fault happened inside table_desc.go.

ajwerner commented 3 years ago

It almost definitely has something to do with schema stuff.

postamar commented 3 years ago

This bug is really scary, to be honest. I'm unable to repro but I'll find a way somehow. Just out of curiosity @rytaft was this a one-off or were you able to trigger this segfault multiple times?

rytaft commented 3 years ago

I only saw this once -- sorry! Thanks for investigating!

postamar commented 3 years ago

I was able to repro it, but just once! Unfortunately my terminal crashed and I didn't copy/paste the output.

Here's what I did after running cockroach demo --nodes 9 --global

ALTER DATABASE movr PRIMARY REGION "us-east1";
ALTER DATABASE movr add REGION "us-west1";
ALTER DATABASE movr add REGION "europe-west1";
ALTER DATABASE movr SURVIVE REGION FAILURE;
alter table rides set locality regional by row;
SHOW JOBS

edit: not quite the same segfault, but it was something else scary related to runtime corruption!

blathers-crl[bot] commented 3 years ago

Hi @postamar, please add branch-* labels to identify which branch(es) this release-blocker affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

ajwerner commented 3 years ago

It sort of feels plausibly related to https://github.com/golang/go/issues/43873 -- but far from certainly.

ajwerner commented 3 years ago

I created https://github.com/cockroachdb/cockroach/pull/62391 to make repro'ing this easier.

awoods187 commented 3 years ago

I crashed demo today by running ./cockroach demo movr --nodes 9 followed by the new refresh built in. Here's a gist https://gist.github.com/awoods187/e9289c706ddc529034368fd3a2a25323

ajwerner commented 3 years ago

@awoods187 this confirms this is a memory corruption bug somewhere and has nothing to do with @postamar and schema code. Nevertheless, it's very scary.

ajwerner commented 3 years ago

@knz one common thread (pun intended) in common here is the syscall in the readline code, though my sense is that that is to be expected. In general when we see memory corruption errors I wonder about cgo things. I don't want to rope you into anything. I appreciate your expertise. If you've got any pointers (pun less intended) on how we might try to better understand this situation, I'd be keen to learn.

knz commented 3 years ago

I'm betting it's somewhat overlapping with #57885

That is, macOS big sur gets libedit confused at some point, which causes the signal handler to not be restored properly upon a call to readline.

I suspect one of the following approaches will trigger the problem:

  1. suspending then restoring the shell with Ctrl+Z / bg
  2. resizing the window
  3. sending one of the other semi-silent signals to the process via the keyboard
knz commented 3 years ago

Also if confirmed to be specific to macOS I don't think this is a GA blocker.

awoods187 commented 3 years ago

I recently upgraded to Big Sur and tried today to test the above by running demo, kill with control c, run demo again and it is definitely hung. In activity monitor, I also see a huge spike in CPU% for cockroach. Closing the shell at this point doesn't end cockroach and I need to pkill cockroach in another window.

andrewwoods@Andrew-W-MacBook-Pro:~/go/src/github.com/cockroachdb/cockroach$ ./cockroach demo movr --nodes 9
#
# Welcome to the CockroachDB demo database!
#
# You are connected to a temporary, in-memory CockroachDB cluster of 9 nodes.
#
# This demo session will attempt to enable enterprise features
# by acquiring a temporary license from Cockroach Labs in the background.
# To disable this behavior, set the environment variable
# COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING=true.
#
# Beginning initialization of the movr dataset, please wait...
knz commented 3 years ago

Andy that particular scenario surprises me. Are you sure the cpu spike is not just the movr dataset initializing normally? Does demo remain hung forever?

awoods187 commented 3 years ago

It was hung for at least ten minutes before I killed it. It doesn't seem deterministic as I tried it again and it didn't hang.

knz commented 3 years ago

thanks for checking.

I wonder if we need to disclaim support for macOS entirely. This is a class of issues that is caused by Apple not playing nice with the open source ecosystem.

The alternative is to deliver a simpler client for macOS users, which does not use advanced line editing for SQL commands.

awoods187 commented 3 years ago

I hit this again today I believe:

demo@127.0.0.1:26257/movr> *
* ERROR: [n1,summaries] a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969
*   | runtime.panicmem
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:212
*   | runtime.sigpanic
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/signal_unix.go:742
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Count
*   |   <autogenerated>:1
*   | github.com/cockroachdb/cockroach/pkg/server/status.extractValue
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:563
*   | github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:595
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:153
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Inspect
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/metric.go:356
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:152
*   | github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:574
*   | github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GenerateNodeStatus
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:455
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:753
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:313
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:752
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:736
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
*   | runtime.goexit
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/asm_amd64.s:1374
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
*
* ERROR: [n1,summaries] a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233
*   | runtime.gopanic
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969
*   | runtime.panicmem
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:212
*   | runtime.sigpanic
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/signal_unix.go:742
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Count
*   |   <autogenerated>:1
*   | github.com/cockroachdb/cockroach/pkg/server/status.extractValue
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:563
*   | github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:595
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:153
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Inspect
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/metric.go:356
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:152
*   | github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:574
*   | github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GenerateNodeStatus
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:455
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:753
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:313
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:752
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:736
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
*   |   /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
*   | runtime.goexit
*   |   /usr/local/Cellar/go/1.15.4/libexec/src/runtime/asm_amd64.s:1374
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x5101caf]

goroutine 1014 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000d16d00, 0x9528720, 0xc00464d710)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x801ed00, 0xbe88070)
    /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000d16d00, 0x9528720, 0xc00464d710)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x801ed00, 0xbe88070)
    /usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Count(0xc0000dbe60, 0x8655f80)
    <autogenerated>:1 +0x2f
github.com/cockroachdb/cockroach/pkg/server/status.extractValue(0x8655f80, 0xc0000dbe60, 0x0, 0x0, 0xc015b75170)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:563 +0x19a
github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue.func1(0xc0009d2810, 0x26, 0x8655f80, 0xc0000dbe60)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:595 +0x191
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each.func1(0x8655f80, 0xc0000dbe60)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:153 +0x6c
github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Inspect(0xc0000dbe60, 0xc04df290a0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/metric.go:356 +0x3c
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each(0xc0010fc440, 0xc04df068a0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/registry.go:152 +0x125
github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue(0xc0010fc440, 0xc04df06890)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:574 +0x65
github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GenerateNodeStatus(0xc000103500, 0x9528720, 0xc00464d710, 0x0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/status/recorder.go:455 +0x62c
github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus.func1(0x9528720, 0xc00464d710)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:753 +0x8b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc000d16d00, 0x9528720, 0xc00464d710, 0x8731684, 0x1a, 0xc015b75e20, 0x0, 0x0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:313 +0xb2
github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus(0xc00165c580, 0x9528720, 0xc00464d710, 0x4a817c800, 0x1, 0x0, 0x0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:752 +0xbd
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x9528720, 0xc00464d710)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:736 +0x16d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000d16d00, 0x9528720, 0xc00464d710, 0x0, 0xc0011bb7c0)
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /Users/andrewwoods/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
knz commented 3 years ago

@awoods187 no this is a different issue. Can you file separately? You're looking at a bug in our implementation of metrics.

Given the stack trace, pleas assign to tobias for triage.

awoods187 commented 3 years ago

Done--thanks @knz https://github.com/cockroachdb/cockroach/issues/63218

knz commented 3 years ago

I removed the "GA-blocker" since this is macOS specific.

Also @erikgrinaker has found out that there is a new change in macOS Big Sur which is poossibly triggering incompatibilities inside the Go runtime (see #63719). This may be the underlying cause here as well.

jseldess commented 3 years ago

I got a panic that might or might not be related, when testing this multi-region local simulation tutorial: https://www.cockroachlabs.com/docs/v21.1/demo-low-latency-multi-region-deployment.html

root@127.0.0.1:26257/movr> *
* ERROR: [n9,s9] a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   /usr/local/go/src/runtime/panic.go:969
*   | runtime.panicmem
*   |   /usr/local/go/src/runtime/panic.go:212
*   | runtime.sigpanic
*   |   /usr/local/go/src/runtime/signal_unix.go:742
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
*   |   /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:951
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
*   |   /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:563
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
*   |   /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:279
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
*   |   /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
*   | runtime.goexit
*   |   /usr/local/go/src/runtime/asm_amd64.s:1374
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
*
* ERROR: [n9,s9] Queued as error 5f5e8ae8ab344fe28313d817ccb9b7f9
*
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
goroutine 2432 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00289c400, 0x9494180, 0xc00563a420)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x7fa7880, 0xbd94420)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick(0xc00590b600, 0x9494180, 0xc008cb0180, 0xc006ed3b30, 0xc008cb0100, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:951 +0x31c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick(0xc00565e000, 0x9494180, 0xc00563a420, 0x1, 0x1)
    /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:563 +0x14a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc003aac1e0, 0x9494180, 0xc00563a420)
    /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:279 +0x309
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00289c400, 0x9494180, 0xc00563a420, 0x0, 0xc003adca00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
knz commented 3 years ago

@jseldess what's the output of version for your executable?

jseldess commented 3 years ago 
~/cockroach-v21.1.0-beta.4.darwin-10.9-amd64$ ./cockroach version
Build Tag:        v21.1.0-beta.4
Build Time:       2021/04/19 15:35:45
Distribution:     CCL
Platform:         darwin amd64 (x86_64-apple-darwin19)
Go Version:       go1.15.10
C Compiler:       Clang 10.0.0
Build Commit ID:  abc4eb5acda343e64415f0267ccd94805d7f2760
Build Type:       release
knz commented 3 years ago

Ok so jesse what you observed is a different issue from what becca observed at the top. Can you also file a different issue? You can ping me on it, it's going to go to the KV project.

Cheers

knz commented 3 years ago

filed #64203 on your behalf.

jseldess commented 3 years ago

Thank you, @knz!

knz commented 3 years ago

I think it's possible the upgrade to the latest go version has alleviated this issue. to folk who've experienced this above, have you seen this happen any time recently with the v21.2 codebase?

rafiss commented 2 years ago

This may have happened again (on 22.1.0-alpha.3): https://github.com/cockroachdb/cockroach/issues/77840

knz commented 1 year ago

We fixed this with the new editor in 23.1