cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test: TestLogic_tenant failed #101420

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test.TestLogic_tenant failed with artifacts on master @ 678cfd4cbebbf3cced16747baf42a4c54cb2a92d:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3d75533]

Stack:

goroutine 66253 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x102?, {0x6564980, 0xc008a1e630})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x49d4b00, 0x8f702d0})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func3.2({0x6564980, 0xc008a1e630})
    github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:360 +0x1d3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b
=== RUN   TestLogic_tenant/drop_tenant
[05:46:47] DROP TENANT dne;
[05:46:47] DROP TENANT IF EXISTS dne;
[05:46:47] CREATE TENANT four;
[05:46:47] SELECT id, active, name
FROM system.tenants WHERE name = 'four'
ORDER BY id;
[05:46:47]  -- OK;
[05:46:47] DROP TENANT four;
[05:46:47] SELECT id, active, name
FROM system.tenants WHERE name = 'four'
ORDER BY id;
[05:46:47]  -- OK;
[05:46:47] SHOW TENANT four;
[05:46:47] CREATE TENANT "five-requiring-quotes";
[05:46:47] DROP TENANT "five-requiring-quotes";
[05:46:47] set default_transaction_read_only = on;;
[05:46:47] DROP TENANT three;
[05:46:47] set default_transaction_read_only = off;;
[05:46:48] DROP TENANT three;
[05:46:48] SHOW TENANTS;
[05:46:48] SHOW TENANT 'two';
[05:46:48] GRANT SYSTEM MANAGETENANT TO testuser;
[05:46:48] SHOW TENANTS;
[05:46:48]  -- OK;
[05:46:48] SHOW TENANT two;
[05:46:48]  -- OK;
[05:46:48] REVOKE SYSTEM MANAGETENANT FROM testuser;
=== RUN   TestLogic_tenant/read_only
[05:46:48] SET default_transaction_read_only = true;
[05:46:48] DROP TENANT three;
[05:46:48] ALTER TENANT three RENAME TO blah;
[05:46:48] CREATE TENANT "read-only";
[05:46:48] SET default_transaction_read_only = false;
=== RUN   TestLogic_tenant/reclaim_name
[05:46:48] CREATE TENANT "to-be-reclaimed";
[05:46:48] DROP TENANT "to-be-reclaimed";
[05:46:48] CREATE TENANT "to-be-reclaimed";
___ Begin jemalloc statistics ___
Version: "0.0.0-0-g000000missing_version_try_git_fetch_tags"
Build-time option settings
  config.cache_oblivious: true
  config.debug: false
  config.fill: true
  config.lazy_lock: false
  config.malloc_conf: ""
  config.opt_safety_checks: false
  config.prof: true
  config.prof_libgcc: true
  config.prof_libunwind: false
  config.stats: true
  config.utrace: false
  config.xmalloc: false
Run-time option settings
  opt.abort: false
  opt.abort_conf: false
  opt.cache_oblivious: true
  opt.confirm_conf: false
  opt.retain: true
  opt.dss: "secondary"
  opt.narenas: 32
  opt.percpu_arena: "disabled"
  opt.oversize_threshold: 8388608
  opt.hpa: false
  opt.hpa_slab_max_alloc: 65536
  opt.hpa_hugification_threshold: 1992294
  opt.hpa_hugify_delay_ms: 10000
  opt.hpa_min_purge_interval_ms: 5000
  opt.hpa_dirty_mult: "0.25"
  opt.hpa_sec_nshards: 4
  opt.hpa_sec_max_alloc: 32768
  opt.hpa_sec_max_bytes: 262144
  opt.hpa_sec_bytes_after_flush: 131072
  opt.hpa_sec_batch_fill_extra: 0
  opt.metadata_thp: "disabled"
  opt.mutex_max_spin: 600
  opt.background_thread: false (background_thread: false)
  opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
  opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
  opt.lg_extent_max_active_fit: 6
  opt.junk: "false"
  opt.zero: false
  opt.experimental_infallible_new: false
  opt.tcache: true
  opt.tcache_max: 32768
  opt.tcache_nslots_small_min: 20
  opt.tcache_nslots_small_max: 200
  opt.tcache_nslots_large: 20
  opt.lg_tcache_nslots_mul: 1
  opt.tcache_gc_incr_bytes: 65536
  opt.tcache_gc_delay_bytes: 0
  opt.lg_tcache_flush_small_div: 1
  opt.lg_tcache_flush_large_div: 1
  opt.thp: "default"
  opt.prof: false
  opt.prof_prefix: "jeprof"
  opt.prof_active: true (prof.active: false)
  opt.prof_thread_active_init: true (prof.thread_active_init: false)
  opt.lg_prof_sample: 19 (prof.lg_sample: 0)
  opt.prof_accum: false
  opt.lg_prof_interval: -1
  opt.prof_gdump: false
  opt.prof_final: false
  opt.prof_leak: false
  opt.prof_leak_error: false
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_interval: -1
  opt.stats_interval_opts: ""
  opt.zero_realloc: "free"
Profiling settings
  prof.thread_active_init: false
  prof.active: false
  prof.gdump: false
  prof.interval: 0
  prof.lg_sample: 0
Arenas: 33
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 14658720, active: 15327232, metadata: 6968480 (n_thp 0), resident: 54321152, mapped: 81223680, retained: 29401088
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 0, num_runs: 0, run_interval: 0 ns
                           n_lock_ops (#/sec)       n_waiting (#/sec)      n_spin_acq (#/sec)  n_owner_switch (#/sec)   total_wait_ns   (#/sec)     max_wait_ns  max_n_thds
background_thread                  44       0               0       0               0       0              13       0               0         0               0           0
max_per_bg_thd                      0       0               0       0               0       0               0       0               0         0               0           0
ctl                            234329    3446               0       0               0       0               4       0               0         0               0           0
prof                                0       0               0       0               0       0               0       0               0         0               0           0
prof_thds_data                      0       0               0       0               0       0               0       0               0         0               0           0
prof_dump                           0       0               0       0               0       0               0       0               0         0               0           0
prof_recent_alloc                   0       0               0       0               0       0               0       0               0         0               0           0
prof_recent_dump                    0       0               0       0               0       0               0       0               0         0               0           0
prof_stats                          0       0               0       0               0       0               0       0               0         0               0           0
Merged arenas stats:
assigned threads: 10
uptime: 68409111911
dss allocation precedence: "N/A"
decaying:  time       npages       sweeps     madvises       purged
   dirty:   N/A         7896            0            1         2049
   muzzy:   N/A            0            0            0            0
                            allocated         nmalloc   (#/sec)         ndalloc   (#/sec)       nrequests   (#/sec)           nfill   (#/sec)          nflush   (#/sec)
small:                        3775648            7666       112            1399        20            1871        27             162         2              45         0
large:                       10883072             259         3             203         2             259         3             259         3               3         0
total:                       14658720            7925       116            1602        23            2130        31             421         6              48         0

active:                      15327232
mapped:                      81223680
retained:                    29401088
base:                         6616224
internal:                      352256
metadata_thp:                       0
tcache_bytes:                 4606208
tcache_stashed_bytes:               0
resident:                    54321152
abandoned_vm:                       0
extent_avail:                      37
                           n_lock_ops (#/sec)       n_waiting (#/sec)      n_spin_acq (#/sec)  n_owner_switch (#/sec)   total_wait_ns   (#/sec)     max_wait_ns  max_n_thds
large                             109       1               0       0               0       0              41       0               0         0               0           0
extent_avail                      869      12               0       0               0       0             182       2               0         0               0           0
extents_dirty                     944      13               0       0               0       0             215       3               0         0               0           0
extents_muzzy                     109       1               0       0               0       0              41       0               0         0               0           0
extents_retained                  376       5               0       0               0       0              83       1               0         0               0           0
decay_dirty                       111       1               0       0               0       0              42       0               0         0               0           0
decay_muzzy                       109       1               0       0               0       0              41       0               0         0               0           0
base                              737      10               0       0               0       0              91       1               0         0               0           0
tcache_list                       119       1               0       0               0       0              50       0               0         0               0           0
hpa_shard                           0       0               0       0               0       0               0       0               0         0               0           0
hpa_shard_grow                      0       0               0       0               0       0               0       0               0         0               0           0
hpa_sec                             0       0               0       0               0       0               0       0               0         0               0           0
bins:           size ind    allocated      nmalloc (#/sec)      ndalloc (#/sec)    nrequests   (#/sec)  nshards      curregs     curslabs  nonfull_slabs regs pgs   util       nfills (#/sec)     nflushes (#/sec)       nslabs     nreslabs (#/sec)      n_lock_ops (#/sec)       n_waiting (#/sec)      n_spin_acq (#/sec)  n_owner_switch (#/sec)   total_wait_ns   (#/sec)     max_wait_ns  max_n_thds
                     ---
                  16   1          416          100       1           74       1            2         0        1           26            1              0  256   1  0.101            1       0            1       0            1            0       0             112       1               0       0               0       0              43       0               0         0               0           0
                  32   2        17184          700      10          163       2            2         0        1          537            7              0  128   1  0.599            7       0            2       0            7            0       0             138       2               0       0               0       0              60       0               0         0               0           0
                  48   3        43200          900      13            0       0            0         0        1          900            9              0  256   3  0.390            9       0            0       0            9            0       0             127       1               0       0               0       0              54       0               0         0               0           0
                  64   4        36864          576       8            0       0            0         0        1          576            9              0   64   1      1            9       0            0       0            9            0       0             127       1               0       0               0       0              55       0               0         0               0           0
                  80   5        56000          700      10            0       0            2         0        1          700            7              0  256   5  0.390            7       0            0       0            7            0       0             123       1               0       0               0       0              53       0               0         0               0           0
                  96   6        84384          900      13           21       0           74         1        1          879            9              0  128   3  0.763            9       0            1       0            9            0       0             128       1               0       0               0       0              57       0               0         0               0           0
                 112   7        22400          200       2            0       0            0         0        1          200            2              0  256   7  0.390            2       0            0       0            2            0       0             113       1               0       0               0       0              43       0               0         0               0           0
                 128   8        36864          288       4            0       0            1         0        1          288            9              0   32   1      1            9       0            0       0            9            0       0             127       1               0       0               0       0              50       0               0         0               0           0
                 160   9        64000          400       5            0       0            0         0        1          400            4              0  128   5  0.781            4       0            0       0            4            0       0             117       1               0       0               0       0              46       0               0         0               0           0
                 192  10        89280          512       7           47       0           12         0        1          465            8              0   64   3  0.908            8       0            1       0            8            0       0             126       1               0       0               0       0              54       0               0         0               0           0
                     ---
                 320  13        92160          384       5           96       1            2         0        1          288            6              0   64   5  0.750            6       0            2       0            6            0       0             123       1               0       0               0       0              48       0               0         0               0           0
                 384  14        12288           32       0            0       0            0         0        1           32            1              0   32   3      1            1       0            0       0            1            0       0             111       1               0       0               0       0              42       0               0         0               0           0
                     ---
                 512  16         5120           10       0            0       0            0         0        1           10            2              0    8   1  0.625            1       0            0       0            2            0       0             112       1               0       0               0       0              42       0               0         0               0           0
                     ---
                 768  18        94464          144       2           21       0           26         0        1          123            9              0   16   3  0.854            9       0            2       0            9            0       0             129       1               0       0               0       0              57       0               0         0               0           0
                 896  19       457856         1440      21          929      13         1737        25        1          511           17              1   32   7  0.939           45       0           30       0           45           11       0             258       3               0       0               0       0              77       1               0         0               0           0
                1024  20        20480           20       0            0       0            0         0        1           20            6              0    4   1  0.833            2       0            0       0            6            0       0             117       1               0       0               0       0              44       0               0         0               0           0
                1280  21        65280           64       0           13       0            1         0        1           51            4              0   16   5  0.796            4       0            2       0            4            0       0             120       1               0       0               0       0              46       0               0         0               0           0
                     ---
                1792  23         7168           16       0           12       0            1         0        1            4            1              0   16   7  0.250            1       0            1       0            1            0       0             112       1               0       0               0       0              42       0               0         0               0           0
                2048  24        20480           10       0            0       0            0         0        1           10            5              0    2   1      1            1       0            0       0            5            0       0             115       1               0       0               0       0              42       0               0         0               0           0
                     ---
                4096  28        40960           10       0            0       0            0         0        1           10           10              0    1   1      1            1       0            0       0           10            0       0             120       1               0       0               0       0              42       0               0         0               0           0
                5120  29       102400           20       0            0       0            0         0        1           20            6              0    4   5  0.833            2       0            0       0            6            0       0             117       1               0       0               0       0              43       0               0         0               0           0
                6144  30        61440           10       0            0       0            0         0        1           10            5              0    2   3      1            1       0            0       0            5            0       0             115       1               0       0               0       0              42       0               0         0               0           0
                7168  31       143360           20       0            0       0            0         0        1           20            6              0    4   7  0.833            2       0            0       0            6            0       0             117       1               0       0               0       0              43       0               0         0               0           0
                8192  32       180224           30       0            8       0            2         0        1           22           22              0    1   2      1            3       0            1       0           30            0       0             143       2               0       0               0       0              44       0               0         0               0           0
               10240  33       542720           60       0            7       0            5         0        1           53           27              0    2   5  0.981            6       0            1       0           30            1       0             146       2               0       0               0       0              48       0               0         0               0           0
               12288  34       761856           70       1            8       0            4         0        1           62           62              0    1   3      1            7       0            1       0           70            0       0             187       2               0       0               0       0              50       0               0         0               0           0
               14336  35       716800           50       0            0       0            0         0        1           50           25              0    2   7      1            5       0            0       0           25            0       0             139       2               0       0               0       0              47       0               0         0               0           0
large:          size ind    allocated      nmalloc (#/sec)      ndalloc (#/sec)    nrequests (#/sec)  curlextents
               16384  36        32768            2       0            0       0            2       0            2
               20480  37        20480            1       0            0       0            1       0            1
               24576  38       147456            6       0            0       0            6       0            6
               28672  39       344064           12       0            0       0           12       0           12
               32768  40       753664           53       0           30       0           53       0           23
               40960  41            0           11       0           11       0           11       0            0
               49152  42            0            1       0            1       0            1       0            0
                     ---
               65536  44        65536            5       0            4       0            5       0            1
               81920  45        81920            1       0            0       0            1       0            1
               98304  46            0            6       0            6       0            6       0            0
              114688  47            0            1       0            1       0            1       0            0
              131072  48            0            1       0            1       0            1       0            0
                     ---
              262144  52      1572864          134       1          128       1          134       1            6
                     ---
              524288  56       524288            6       0            5       0            6       0            1
                     ---
             1048576  60      1048576            6       0            5       0            6       0            1
                     ---
             2097152  64      2097152            6       0            5       0            6       0            1
                     ---
             4194304  68      4194304            6       0            5       0            6       0            1
                     ---
             8388608  72            0            1       0            1       0            1       0            0
                     ---
extents:        size ind       ndirty        dirty       nmuzzy        muzzy    nretained     retained       ntotal        total
                4096   0            2         8192            0            0            1         4096            3        12288
                8192   1            9        73728            0            0            0            0            9        73728
               12288   2            9       110592            0            0            0            0            9       110592
                     ---
               20480   4            2        40960            0            0            0            0            2        40960
               24576   5            1        24576            0            0            0            0            1        24576
               28672   6            2        57344            0            0            0            0            2        57344
                     ---
               40960   8            4       147456            0            0            0            0            4       147456
               49152   9            4       188416            0            0            1        49152            5       237568
               57344  10            1        57344            0            0            0            0            1        57344
                     ---
               81920  12            2       155648            0            0            0            0            2       155648
               98304  13            1        94208            0            0            1        90112            2       184320
              114688  14            2       212992            0            0            1       114688            3       327680
              131072  15            2       245760            0            0            1       126976            3       372736
              163840  16            2       294912            0            0            2       294912            4       589824
                     ---
              229376  18            2       450560            0            0            0            0            2       450560
              262144  19            1       233472            0            0            0            0            1       233472
              327680  20            8      2183168            0            0            0            0            8      2183168
                     ---
              524288  23            3      1507328            0            0            3      1560576            6      3067904
              655360  24            2      1122304            0            0            0            0            2      1122304
                     ---
              917504  26            2      1605632            0            0            0            0            2      1605632
             1048576  27            2      2011136            0            0            6      6180864            8      8192000
             1310720  28            0            0            0            0            1      1187840            1      1187840
             1572864  29            1      1564672            0            0            3      4636672            4      6201344
             1835008  30            1      1585152            0            0            1      1789952            2      3375104
                     ---
             2621440  32            1      2101248            0            0            0            0            1      2101248
             3145728  33            0            0            0            0            1      2879488            1      2879488
             3670016  34            1      3670016            0            0            0            0            1      3670016
                     ---
             5242880  36            3     12595200            0            0            0            0            3     12595200
                     ---
            10485760  40            0            0            0            0            1     10485760            1     10485760
                     ---
Bytes in small extent cache: 0
HPA shard stats:
  Purge passes: 0 (0 / sec)
  Purges: 0 (0 / sec)
  Hugeifies: 0 (0 / sec)
  Dehugifies: 0 (0 / sec)
Log preceding fatal error

``` [05:46:46] ALTER TENANT [4] RENAME TO blux; [05:46:46] ALTER TENANT blux RENAME TO 'blix'; [05:46:47] SELECT * FROM [SHOW TENANTS] WHERE id = 4; [05:46:47] -- OK; [05:46:47] ALTER TENANT blix RENAME TO three; [05:46:47] SELECT * FROM [SHOW TENANTS] WHERE id = 4; [05:46:47] -- OK; [05:46:47] SELECT id, name, data_state, service_mode, source_tenant_name FROM [SHOW TENANTS WITH REPLICATION STATUS] WHERE id = 4; [05:46:47] -- OK; [05:46:47] SHOW TENANT seven; [05:46:47] CREATE TENANT three; [05:46:47] CREATE TENANT IF NOT EXISTS three; [05:46:47] set default_transaction_read_only = on;; [05:46:47] CREATE TENANT four;; [05:46:47] set default_transaction_read_only = off;; [05:46:47] CREATE TENANT four; === RUN TestLogic_tenant/service_mode [05:46:49] ALTER TENANT 'tenant-one' START SERVICE EXTERNAL; [05:46:49] ALTER TENANT 'tenant-one' START SERVICE EXTERNAL; [05:46:49] ALTER TENANT two START SERVICE SHARED; [05:46:49] ALTER TENANT two START SERVICE SHARED; [05:46:49] ALTER TENANT 'tenant-one' START SERVICE SHARED; [05:46:49] ALTER TENANT two START SERVICE EXTERNAL; [05:46:49] DROP TENANT two; [05:46:49] DROP TENANT 'tenant-one'; [05:46:49] SHOW TENANTS; [05:46:49] -- OK; [05:46:49] ALTER TENANT two STOP SERVICE; [05:46:49] ALTER TENANT 'tenant-one' STOP SERVICE; [05:46:49] SHOW TENANTS; [05:46:49] -- OK; [05:46:49] DROP TENANT two; [05:46:49] DROP TENANT 'tenant-one'; === RUN TestLogic_tenant/tenant_templates [05:46:50] SHOW CLUSTER SETTING sql.create_tenant.default_template; [05:46:50] -- OK; [05:46:50] CREATE TENANT othertenant LIKE system; [05:46:50] CREATE TENANT tmpl; ALTER TENANT tmpl GRANT CAPABILITY can_view_node_info; -- will be copied ALTER TENANT tmpl SET CLUSTER SETTING trace.debug.enable = true; -- will be copied -- Simulate resource limits. Will be copied. -- Note: we cannot use the update_tenant_resource_limits() builtin -- directly here because it can only be used from a CCL binary. INSERT INTO system.tenant_usage( tenant_id, instance_id, next_instance_id, last_update, ru_burst_limit, ru_refill_rate, ru_current, current_share_sum, total_consumption) VALUES ((SELECT id FROM system.tenants WHERE name = 'tmpl'), 0, 0, now(), 11, 22, 33, 44, ''::BYTES); ALTER TENANT tmpl START SERVICE SHARED; -- will not be copied.; [05:46:50] CREATE TENANT othertenant LIKE tmpl; ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-26938

cucaroach commented 1 year ago

Dupe of #101403