search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9k stars 1.07k forks source link

[DocDB][LST] Master crash after Colocated Tablet not found for table #12988

Closed def- closed 2 years ago

def- commented 2 years ago

Jira Link: DB-2716

Description

I initially thought this is another instance of https://github.com/yugabyte/yugabyte-db/issues/12813, but after talking to @spolitov this appears to be a separate bug:

$ cd ~/code/yugabyte-db
$ git checkout b0dcb81db7f0793ab7e6278ec1f2e34d344d3a7c
$ ./yb_build.sh release
$ bin/yb-ctl --replication_factor 1 create --tserver_flags=ysql_enable_packed_row=true,ysql_packed_row_size_limit=1700 --master_flags=ysql_enable_packed_row=true,ysql_packed_row_size_limit=1700
$ cd ~/code/yb-long-system-test
$ python3.9 ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
2022-06-22 10:19:38,821 MainThread INFO
2022-06-22 10:19:38,821 MainThread INFO     --------------------------------------------------------------------------------
2022-06-22 10:19:38,821 MainThread INFO     Running Long System Test 0.1
2022-06-22 10:19:38,821 MainThread INFO     --------------------------------------------------------------------------------
2022-06-22 10:19:38,821 MainThread INFO
2022-06-22 10:19:39,105 MainThread INFO     Reproduce with: git checkout 6fdda644 && ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
2022-06-22 10:19:39,625 MainThread INFO     Database version: PostgreSQL 11.2-YB-2.15.1.0-b0 on x86_64-pc-linux-gnu, compiled by clang version 12.0.1 (https://github.com/yugabyte/llvm-project.git bdb147e675d8c87cee72cc1f87c4b82855977d94), 64-bit
2022-06-22 10:19:39,628 MainThread INFO     Creating tables for database db_lst_032216
2022-06-22 10:19:51,230 MainThread INFO     Starting worker_0: RandomSelectAction, SetConfigAction
2022-06-22 10:19:51,231 MainThread INFO     Starting worker_1: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-22 10:19:51,232 MainThread INFO     Starting worker_2: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-22 10:19:51,233 MainThread INFO     Starting worker_3: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-22 10:19:51,234 MainThread INFO     Starting worker_4: RandomSelectAction, SetConfigAction
2022-06-22 10:19:51,234 MainThread INFO     Starting worker_5: SingleInsertAction, SingleUpdateAction, SingleDeleteAction, BulkInsertAction, BulkUpdateAction, SetConfigAction
2022-06-22 10:19:51,235 MainThread INFO     Starting worker_6: RandomSelectAction, SetConfigAction
2022-06-22 10:19:51,235 MainThread INFO     Starting worker_7: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-22 10:19:51,236 MainThread INFO     Starting worker_8: SingleInsertAction, SingleUpdateAction, SingleDeleteAction, BulkInsertAction, BulkUpdateAction, SetConfigAction
2022-06-22 10:19:51,237 MainThread INFO     Starting worker_9: RandomSelectAction, SetConfigAction
2022-06-22 10:20:01,247 MainThread INFO     Worker queries/s: [012.8][001.0][001.4][000.7][019.7][016.3][017.8][000.5][004.7][001.5]
[...]
2022-06-22 10:55:43,462 MainThread INFO     Worker queries/s: [002.7][000.5][000.7][000.4][003.6][006.5][002.9][000.4][008.8][002.3]
2022-06-22 10:55:53,472 MainThread INFO     Worker queries/s: [001.4][000.7][000.2][000.4][002.3][004.9][003.4][000.3][004.2][000.9]
2022-06-22 10:56:03,482 MainThread INFO     Worker queries/s: [002.1][000.5][000.2][000.3][002.9][006.5][003.1][000.8][005.6][003.6]
2022-06-22 10:56:13,492 MainThread INFO     Worker queries/s: [000.1][000.0][000.0][000.0][000.1][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:56:23,496 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:56:33,499 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:56:43,509 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:56:53,520 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:03,530 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:13,540 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:23,551 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:33,561 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:43,571 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:57:53,577 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:03,587 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:13,596 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:23,606 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:33,616 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:43,624 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:58:53,634 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:03,645 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:13,655 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:23,666 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:33,676 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:43,686 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 10:59:53,689 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:03,690 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:13,701 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:23,704 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:33,713 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:43,723 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:00:53,733 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:01:03,744 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
2022-06-22 11:01:05,532 worker_4   ERROR    Unexpected query failure: OperationalError
Query: EXPLAIN (ANALYZE TRUE, VERBOSE FALSE, COSTS FALSE, BUFFERS TRUE, TIMING FALSE, SUMMARY FALSE, FORMAT JSON)
SELECT -21, numrange((-85.87973200454147), NULL), numrange(NULL, (70.58518724830691)), CAST((2) AS FLOAT4), -2147483648, '1900-10-20', -35, '[2147483647,)'::INT4RANGE, 35, -100, count(tg4_0.c10_bigint), '1998-08-31', '[6.8111913471356615,97.57575546402867]'::NUMRANGE, 57.46201309998281, '{"a": 4, "b": ["0", "1", "2", "3", "4", "5", "6", "7", "8"], "c": false}'::json, 'empty'::INT4RANGE, count(*), CAST((52.70681718532768) AS NUMERIC), -99.17234933165086, CAST((-7) AS NUMERIC), -2147483648, (range_merge((('[-26,)'::INT2RANGE) * ('(-33,-16)'::INT2RANGE)), (('[-83,)'::INT2RANGE) * ((range_merge((('(-16,84)'::INT2RANGE) * ((('(20,68]'::INT2RANGE) * (('(,18)'::INT2RANGE) * ('(98,100]'::INT2RANGE))) * ('[-79,43]'::INT2RANGE))), (range_merge(('(-85,-74)'::INT2RANGE), (('(-66,92)'::INT2RANGE) * (('(-81,-19)'::INT2RANGE) * (range_merge(('(,69)'::INT2RANGE), ('[55,70]'::INT2RANGE))))))))) * ('(,-87)'::INT2RANGE))))) -|- ('[-48,)'::INT2RANGE), int8range((28), NULL), '{"a": 7, "b": ["0", "1", "2", "3", "4"], "c": false}'::json, '1956-05-10', int4range((-94), NULL), range_merge((range_merge(('(-91,87]'::INT2RANGE), ('[-87,-75]'::INT2RANGE))), ('(,-72)'::INT2RANGE)) FROM tg4_0 WHERE TRUE ORDER BY 13 DESC OFFSET 12;
  values: None
  runtime: 2022-06-22 10:56:03.486 - 2022-06-22 11:01:05.532
  supports explain: False
  supports rollback: False
  affected rows: None
Action: RandomSelectAction
Error class: OperationalError
Error code: None
Error message: FATAL:  Could not reconnect to database
HINT:  Database might have been dropped by another user
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
Transaction isolation level: default
DB Node: host: 127.0.0.1, port: 5433
DB Backend PID: 427570

Can't connect to database anymore after this:

[dfelsing@dev-server-dfelsing yb-long-system-test]$ python3.9 ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
2022-06-22 11:26:20,174 MainThread INFO
2022-06-22 11:26:20,174 MainThread INFO     --------------------------------------------------------------------------------
2022-06-22 11:26:20,174 MainThread INFO     Running Long System Test 0.1
2022-06-22 11:26:20,174 MainThread INFO     --------------------------------------------------------------------------------
2022-06-22 11:26:20,174 MainThread INFO
2022-06-22 11:26:20,181 MainThread INFO     Reproduce with: git checkout 6fdda644 && ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
Traceback (most recent call last):
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/./long_system_test.py", line 279, in <module>
    main()
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/./long_system_test.py", line 270, in main
    lst.run()
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/./long_system_test.py", line 203, in run
    self._create_tables()
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/./long_system_test.py", line 166, in _create_tables
    conn = psycopg2.connect(**self._get_connect_opts())
  File "/usr/local/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "127.0.0.1", port 5433 failed: timeout expired
def- commented 2 years ago 
[dfelsing@dev-server-dfelsing yugabyte-data]$ coredumpctl gdb 369052
           PID: 369052 (yb-master)
           UID: 2200 (dfelsing)
           GID: 1500 (ybdev)
        Signal: 11 (SEGV)
     Timestamp: Wed 2022-06-22 10:56:03 UTC (15min ago)
  Command Line: /nfusr/dev-server/dfelsing/code/yugabyte-db/build/latest/bin/yb-master --fs_data_dirs /nfusr/dev-server/dfelsing/yugabyte-data/node-1/disk-1 --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/nfusr/dev-server/dfelsing/code/yugabyte-db/build/release-clang12-linuxbrew-dynamic-ninja --replication_factor=1 --yb_num_shards_per_tserver 2 --ysql_num_shards_per_tserver=2 --default_memory_limit_to_ram_ratio=0.35 --master_addresses 127.0.0.1:7100 --enable_ysql=true --ysql_enable_packed_row=true --ysql_packed_row_size_limit=1700
    Executable: /nfusr/dev-server/dfelsing/code/yugabyte-db/build/release-clang12-linuxbrew-dynamic-ninja/bin/yb-master
 Control Group: /
         Slice: -.slice
       Boot ID: 6de03a0d3b4d45a39be8dd54cb2c301d
    Machine ID: 8ce570b287c573260282d2e8edd33beb
      Hostname: dev-server-dfelsing
       Storage: /var/lib/systemd/coredump/core.yb-master.2200.6de03a0d3b4d45a39be8dd54cb2c301d.369052.1655895363000000.lz4
       Message: Process 369052 (yb-master) of user 2200 dumped core.

                Stack trace of thread 400893:
                #0  0x00007f1074ceb7fe _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_ (libc++.so.1)
                #1  0x00007f10792ec90d _ZN2yb6master26AsyncRemoveTableFromTabletC1EPNS0_6MasterEPNS_10ThreadPoolERK13scoped_refptrINS0_10TabletInfoEERKS6_INS0_9TableInfoEE (libmaster.so)
                #2  0x00007f10793d0ce0 _ZNSt3__115allocate_sharedIN2yb6master26AsyncRemoveTableFromTabletENS_9allocatorIS3_EEJRPNS2_6MasterEPNS1_10ThreadPoolE13scoped_refptrINS2_10TabletInfoEERKSB_INS2_9TableInfoEEEvEENS_10shared_ptrIT_EERKT0_DpOT1_ (libmaster.so)
                #3  0x00007f1079367596 _ZN2yb6master14CatalogManager19DeleteTableInternalEPKNS0_20DeleteTableRequestPBEPNS0_21DeleteTableResponsePBEPNS_3rpc10RpcContextE (libmaster.so)
                #4  0x00007f10793667ff _ZN2yb6master14CatalogManager11DeleteTableEPKNS0_20DeleteTableRequestPBEPNS0_21DeleteTableResponsePBEPNS_3rpc10RpcContextE (libmaster.so)
                #5  0x00007f1079461950 _ZN2yb6master17MasterServiceBase14HandleOnLeaderINS0_20DeleteTableRequestPBENS0_21DeleteTableResponsePBEZNS1_8HandleInINS0_14CatalogManagerES3_S4_EEvPKT0_PT1_PNS_3rpc10RpcContextEMT_FNS_6StatusES9_SB_SE_EPKciSK_NS_17StronglyTypedBoolINS0_19HoldCatalogLock_TagEEEEUlvE_EEvPKSF_PS7_SE_SA_SK_iSK_SN_ (libmaster.so)
                #6  0x00007f107945e90a _ZN2yb6master12_GLOBAL__N_120MasterDdlServiceImpl11DeleteTableEPKNS0_20DeleteTableRequestPBEPNS0_21DeleteTableResponsePBENS_3rpc10RpcContextE (libmaster.so)
                #7  0x00007f1076dd46d4 _ZNSt3__110__function6__funcIZN2yb6master11MasterDdlIf11InitMethodsERK13scoped_refptrINS2_12MetricEntityEEE4$_11NS_9allocatorISA_EEFvNS_10shared_ptrINS2_3rpc11InboundCallEEEEEclEOSG_ (libmaster_proto.so)
                #8  0x00007f1076dcfca6 _ZN2yb6master11MasterDdlIf6HandleENSt3__110shared_ptrINS_3rpc11InboundCallEEE (libmaster_proto.so)
                #9  0x00007f10764d7333 _ZN2yb3rpc15ServicePoolImpl6HandleENSt3__110shared_ptrINS0_11InboundCallEEE (libyrpc.so)
                #10 0x00007f1076480d55 _ZN2yb3rpc11InboundCall15InboundCallTask3RunEv (libyrpc.so)
                #11 0x00007f10764e32c0 _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #12 0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #13 0x00007f107450a694 start_thread (libpthread.so.0)
                #14 0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369073:
                #0  0x00007f107450f3b8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1076017a76 _ZNK2yb17ConditionVariable9WaitUntilERKNS_8MonoTimeE (libyb_util.so)
                #2  0x00007f1079861606 n/a (ld.so)
                #3  0x00007f1079860f69 n/a (ld.so)
                #4  0x00007f1076135aba _ZN2yb10ThreadPool14DispatchThreadEb (libyb_util.so)
                #5  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #6  0x00007f107450a694 start_thread (libpthread.so.0)
                #7  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369120:
                #0  0x00007f1078b9c810 _ZN2yb5docdb11SeekForwardERKNS_5SliceEPN7rocksdb8IteratorE (libyb_docdb.so)
                #1  0x00007f1078bdea87 _ZN2yb5docdb19IntentAwareIterator18SeekForwardRegularERKNS_5SliceE (libyb_docdb.so)
                #2  0x00007f1078bde8cf _ZN2yb5docdb19IntentAwareIterator11SeekForwardEPNS0_8KeyBytesE (libyb_docdb.so)
                #3  0x00007f1078bde671 _ZN2yb5docdb19IntentAwareIterator11SeekForwardERKNS_5SliceE (libyb_docdb.so)
                #4  0x00007f1078b91027 _ZN2yb5docdb16DocDBTableReader9GetHelper20SeekProjectionColumnEv (libyb_docdb.so)
                #5  0x00007f1078b8ea6d _ZN2yb5docdb16DocDBTableReader9GetHelper14DoHandleRecordERKNS0_14FetchKeyResultERKNS_5SliceENS_17StronglyTypedBoolINS0_12_GLOBAL__N_118CheckExistOnly_TagEEE (libyb_docdb.so)
                #6  0x00007f1078b8e0b5 _ZN2yb5docdb16DocDBTableReader9GetHelper4ScanENS_17StronglyTypedBoolINS0_12_GLOBAL__N_118CheckExistOnly_TagEEE (libyb_docdb.so)
                #7  0x00007f1078b8fbd3 _ZN2yb5docdb16DocDBTableReader9GetHelper3RunEv (libyb_docdb.so)
                #8  0x00007f1078b8d39d _ZN2yb5docdb16DocDBTableReader3GetERKNS_5SliceEPNS0_11SubDocumentE (libyb_docdb.so)
                #9  0x00007f1078bbfa30 _ZNK2yb5docdb18DocRowwiseIterator7HasNextEv (libyb_docdb.so)
                #10 0x00007f1078bc1643 _ZN2yb5docdb18DocRowwiseIterator9SeekTupleERKNS_5SliceE (libyb_docdb.so)
                #11 0x00007f1078bfcc19 _ZN2yb5docdb18PgsqlReadOperation13ExecuteScalarERKNS0_12YQLStorageIfENSt3__16chrono10time_pointINS_15CoarseMonoClockENS6_8durationIxNS5_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS0_14DocReadContextEPSI_PNS_10faststringEPNS_10HybridTimeEPb (libyb_docdb.so)
                #12 0x00007f1078bfa7ea _ZN2yb5docdb18PgsqlReadOperation7ExecuteERKNS0_12YQLStorageIfENSt3__16chrono10time_pointINS_15CoarseMonoClockENS6_8durationIxNS5_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS0_14DocReadContextEPSI_PNS_10faststringEPNS_10HybridTimeE (libyb_docdb.so)
                #13 0x00007f1078d3b52b _ZN2yb6tablet14AbstractTablet23ProcessPgsqlReadRequestENSt3__16chrono10time_pointINS_15CoarseMonoClockENS3_8durationIxNS2_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS_18PgsqlReadRequestPBERKNS2_10shared_ptrINS0_9TableInfoEEERKNS_27TransactionOperationContextEPNS0_22PgsqlReadRequestResultEPm (libtablet.so)
                #14 0x00007f1078d75aa9 _ZN2yb6tablet6Tablet22HandlePgsqlReadRequestENSt3__16chrono10time_pointINS_15CoarseMonoClockENS3_8durationIxNS2_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS_18PgsqlReadRequestPBERKNS_21TransactionMetadataPBERKNS_24SubTransactionMetadataPBEPNS0_22PgsqlReadRequestResultEPm (libtablet.so)
                #15 0x00007f1078fb388e _ZN2yb7tserver12_GLOBAL__N_19ReadQuery8CompleteEv (libtserver.so)
                #16 0x00007f1078fb17b2 _ZN2yb7tserver11PerformReadEPNS0_14TabletServerIfEPNS0_18ReadTabletProviderEPKNS0_13ReadRequestPBEPNS0_14ReadResponsePBENS_3rpc10RpcContextE (libtserver.so)
                #17 0x00007f1079006525 _ZN2yb7tserver17TabletServiceImpl4ReadEPKNS0_13ReadRequestPBEPNS0_14ReadResponsePBENS_3rpc10RpcContextE (libtserver.so)
                #18 0x00007f1077a877ee _ZNSt3__110__function6__funcIZN2yb7tserver21TabletServerServiceIf11InitMethodsERK13scoped_refptrINS2_12MetricEntityEEE3$_1NS_9allocatorISA_EEFvNS_10shared_ptrINS2_3rpc11InboundCallEEEEEclEOSG_ (libtserver_service_proto.so)
                #19 0x00007f1077a86e06 _ZN2yb7tserver21TabletServerServiceIf6HandleENSt3__110shared_ptrINS_3rpc11InboundCallEEE (libtserver_service_proto.so)
                #20 0x00007f10764d7333 _ZN2yb3rpc15ServicePoolImpl6HandleENSt3__110shared_ptrINS0_11InboundCallEEE (libyrpc.so)
                #21 0x00007f1076480d55 _ZN2yb3rpc11InboundCall15InboundCallTask3RunEv (libyrpc.so)
                #22 0x00007f10764e32c0 _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #23 0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #24 0x00007f107450a694 start_thread (libpthread.so.0)
                #25 0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369074:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10761358b3 _ZN2yb10ThreadPool14DispatchThreadEb (libyb_util.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369082:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10764856fa _ZN5boost4asio6detail9scheduler10do_run_oneERNS1_27conditionally_enabled_mutex11scoped_lockERNS1_21scheduler_thread_infoERKNS_6system10error_codeE (libyrpc.so)
                #2  0x00007f1076485311 _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE (libyrpc.so)
                #3  0x00007f1076482485 _ZN2yb3rpc12IoThreadPool4Impl7ExecuteEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369087:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400908:
                #0  0x00007f1078bbf465 _ZNK2yb5docdb18DocRowwiseIterator7HasNextEv (libyb_docdb.so)
                #1  0x00007f1078bfcac0 _ZN2yb5docdb18PgsqlReadOperation13ExecuteScalarERKNS0_12YQLStorageIfENSt3__16chrono10time_pointINS_15CoarseMonoClockENS6_8durationIxNS5_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS0_14DocReadContextEPSI_PNS_10faststringEPNS_10HybridTimeEPb (libyb_docdb.so)
                #2  0x00007f1078bfa7ea _ZN2yb5docdb18PgsqlReadOperation7ExecuteERKNS0_12YQLStorageIfENSt3__16chrono10time_pointINS_15CoarseMonoClockENS6_8durationIxNS5_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS0_14DocReadContextEPSI_PNS_10faststringEPNS_10HybridTimeE (libyb_docdb.so)
                #3  0x00007f1078d3b52b _ZN2yb6tablet14AbstractTablet23ProcessPgsqlReadRequestENSt3__16chrono10time_pointINS_15CoarseMonoClockENS3_8durationIxNS2_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS_18PgsqlReadRequestPBERKNS2_10shared_ptrINS0_9TableInfoEEERKNS_27TransactionOperationContextEPNS0_22PgsqlReadRequestResultEPm (libtablet.so)
                #4  0x00007f1078d75aa9 _ZN2yb6tablet6Tablet22HandlePgsqlReadRequestENSt3__16chrono10time_pointINS_15CoarseMonoClockENS3_8durationIxNS2_5ratioILl1ELl1000000000EEEEEEERKNS_14ReadHybridTimeEbRKNS_18PgsqlReadRequestPBERKNS_21TransactionMetadataPBERKNS_24SubTransactionMetadataPBEPNS0_22PgsqlReadRequestResultEPm (libtablet.so)
                #5  0x00007f1078fb388e _ZN2yb7tserver12_GLOBAL__N_19ReadQuery8CompleteEv (libtserver.so)
                #6  0x00007f1078fb17b2 _ZN2yb7tserver11PerformReadEPNS0_14TabletServerIfEPNS0_18ReadTabletProviderEPKNS0_13ReadRequestPBEPNS0_14ReadResponsePBENS_3rpc10RpcContextE (libtserver.so)
                #7  0x00007f1079006525 _ZN2yb7tserver17TabletServiceImpl4ReadEPKNS0_13ReadRequestPBEPNS0_14ReadResponsePBENS_3rpc10RpcContextE (libtserver.so)
                #8  0x00007f1077a877ee _ZNSt3__110__function6__funcIZN2yb7tserver21TabletServerServiceIf11InitMethodsERK13scoped_refptrINS2_12MetricEntityEEE3$_1NS_9allocatorISA_EEFvNS_10shared_ptrINS2_3rpc11InboundCallEEEEEclEOSG_ (libtserver_service_proto.so)
                #9  0x00007f1077a86e06 _ZN2yb7tserver21TabletServerServiceIf6HandleENSt3__110shared_ptrINS_3rpc11InboundCallEEE (libtserver_service_proto.so)
                #10 0x00007f10764d7333 _ZN2yb3rpc15ServicePoolImpl6HandleENSt3__110shared_ptrINS0_11InboundCallEEE (libyrpc.so)
                #11 0x00007f1076480d55 _ZN2yb3rpc11InboundCall15InboundCallTask3RunEv (libyrpc.so)
                #12 0x00007f10764e32c0 _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #13 0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #14 0x00007f107450a694 start_thread (libpthread.so.0)
                #15 0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369052:
                #0  0x00007f107451246d __nanosleep (libpthread.so.0)
                #1  0x00007f1074cf0b0b _ZNSt3__111this_thread9sleep_forERKNS_6chrono8durationIxNS_5ratioILl1ELl1000000000EEEEE (libc++.so.1)
                #2  0x00007f1077b7fe6f _ZN2yb6server15TotalMemWatcher20MemoryMonitoringLoopENSt3__18functionIFvvEEENS3_IFbvEEE (libserver_process.so)
                #3  0x0000000000205a44 main (yb-master)
                #4  0x00007f1074185825 __libc_start_main (libc.so.6)
                #5  0x00000000002050a9 _start (yb-master)

                Stack trace of thread 400904:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369076:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10761358b3 _ZN2yb10ThreadPool14DispatchThreadEb (libyb_util.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369135:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f107773a6bb _ZN7rocksdb10ThreadPool8BGThreadEm (librocksdb.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 374879:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400902:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400901:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369072:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10761358b3 _ZN2yb10ThreadPool14DispatchThreadEb (libyb_util.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 434645:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f107610171d _ZN2yb12_GLOBAL__N_124PriorityThreadPoolWorker3RunEv (libyb_util.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369407:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369084:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10764856fa _ZN5boost4asio6detail9scheduler10do_run_oneERNS1_27conditionally_enabled_mutex11scoped_lockERNS1_21scheduler_thread_infoERKNS_6system10error_codeE (libyrpc.so)
                #2  0x00007f1076485311 _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE (libyrpc.so)
                #3  0x00007f1076482485 _ZN2yb3rpc12IoThreadPool4Impl7ExecuteEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369811:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400906:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369133:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f107773a6bb _ZN7rocksdb10ThreadPool8BGThreadEm (librocksdb.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400905:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400896:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369081:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f10764856fa _ZN5boost4asio6detail9scheduler10do_run_oneERNS1_27conditionally_enabled_mutex11scoped_lockERNS1_21scheduler_thread_infoERKNS_6system10error_codeE (libyrpc.so)
                #2  0x00007f1076485311 _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE (libyrpc.so)
                #3  0x00007f1076482485 _ZN2yb3rpc12IoThreadPool4Impl7ExecuteEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369141:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f10764846fe _ZN5boost4asio6detail13epoll_reactor3runElRNS1_8op_queueINS1_19scheduler_operationEEE (libyrpc.so)
                #2  0x00007f10764855fc _ZN5boost4asio6detail9scheduler10do_run_oneERNS1_27conditionally_enabled_mutex11scoped_lockERNS1_21scheduler_thread_infoERKNS_6system10error_codeE (libyrpc.so)
                #3  0x00007f1076485311 _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE (libyrpc.so)
                #4  0x00007f1076482485 _ZN2yb3rpc12IoThreadPool4Impl7ExecuteEv (libyrpc.so)
                #5  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #6  0x00007f107450a694 start_thread (libpthread.so.0)
                #7  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400903:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369083:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f10764846fe _ZN5boost4asio6detail13epoll_reactor3runElRNS1_8op_queueINS1_19scheduler_operationEEE (libyrpc.so)
                #2  0x00007f10764855fc _ZN5boost4asio6detail9scheduler10do_run_oneERNS1_27conditionally_enabled_mutex11scoped_lockERNS1_21scheduler_thread_infoERKNS_6system10error_codeE (libyrpc.so)
                #3  0x00007f1076485311 _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE (libyrpc.so)
                #4  0x00007f1076482485 _ZN2yb3rpc12IoThreadPool4Impl7ExecuteEv (libyrpc.so)
                #5  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #6  0x00007f107450a694 start_thread (libpthread.so.0)
                #7  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369085:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400895:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369090:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369129:
                #0  0x00007f107450f3b8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce47eb _ZNSt3__118condition_variable15__do_timed_waitERNS_11unique_lockINS_5mutexEEENS_6chrono10time_pointINS5_12system_clockENS5_8durationIxNS_5ratioILl1ELl1000000000EEEEEEE (libc++.so.1)
                #2  0x00007f1076055fc0 _ZN2yb12_GLOBAL__N_126LongOperationTrackerHelper7ExecuteEv (libyb_util.so)
                #3  0x00007f107605655d _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN2yb12_GLOBAL__N_126LongOperationTrackerHelperEFvvEJPSA_EEEEEEEEPvSG_ (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369092:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369097:
                #0  0x00007f107450f3b8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce47eb _ZNSt3__118condition_variable15__do_timed_waitERNS_11unique_lockINS_5mutexEEENS_6chrono10time_pointINS5_12system_clockENS5_8durationIxNS_5ratioILl1ELl1000000000EEEEEEE (libc++.so.1)
                #2  0x00007f1078e3d98a _ZN2yb18MaintenanceManager18RunSchedulerThreadEv (libtablet.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400891:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400907:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369109:
                #0  0x00007f107450f3b8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1076017a76 _ZNK2yb17ConditionVariable9WaitUntilERKNS_8MonoTimeE (libyb_util.so)
                #2  0x00007f1076017d37 _ZNK2yb14CountDownLatch9WaitUntilENS_8MonoTimeE (libyb_util.so)
                #3  0x00007f1076017db2 _ZNK2yb14CountDownLatch7WaitForENS_9MonoDeltaE (libyb_util.so)
                #4  0x00007f1078fca710 _ZN2yb7tserver26RemoteBootstrapServiceImpl18EndExpiredSessionsEv (libtserver.so)
                #5  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #6  0x00007f107450a694 start_thread (libpthread.so.0)
                #7  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369112:
                #0  0x00007f10742425cd poll (libc.so.6)
                #1  0x00007f1077b9639b master_thread (libserver_process.so)
                #2  0x00007f107450a694 start_thread (libpthread.so.0)
                #3  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369138:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f1076011c6b _ZN2yb14BackgroundTask10WaitForJobEv (libyb_util.so)
                #3  0x00007f1076011871 _ZN2yb14BackgroundTask3RunEv (libyb_util.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369115:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f1076466f77 _ZN2yb3rpc8Acceptor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369127:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369130:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369132:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f107773a6bb _ZN7rocksdb10ThreadPool8BGThreadEm (librocksdb.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369139:
                #0  0x00007f107450f3b8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1076017b1b _ZNK2yb17ConditionVariable9TimedWaitERKNS_9MonoDeltaE (libyb_util.so)
                #2  0x00007f10793ff9f1 _ZN2yb6master21CatalogManagerBgTasks4WaitEi (libmaster.so)
                #3  0x00007f10793ffd40 _ZN2yb6master21CatalogManagerBgTasks3RunEv (libmaster.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369134:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f107773a6bb _ZN7rocksdb10ThreadPool8BGThreadEm (librocksdb.so)
                #2  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #3  0x00007f107450a694 start_thread (libpthread.so.0)
                #4  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369405:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369810:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369968:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 371512:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 371514:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 372533:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 380987:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400894:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400900:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 380985:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369406:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369088:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369188:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 380986:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 400897:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 371269:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369091:
                #0  0x00007f107424b9f3 epoll_wait (libc.so.6)
                #1  0x00007f1075e09db8 epoll_poll (libev.so.4)
                #2  0x00007f1075e0512e ev_run (libev.so.4)
                #3  0x00007f10764a210c _ZN2yb3rpc7Reactor9RunThreadEv (libyrpc.so)
                #4  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #5  0x00007f107450a694 start_thread (libpthread.so.0)
                #6  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369809:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 369404:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

                Stack trace of thread 371513:
                #0  0x00007f107450f00f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f1074ce4752 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1)
                #2  0x00007f10764e327d _ZN2yb3rpc12_GLOBAL__N_16Worker7ExecuteEv (libyrpc.so)
                #3  0x00007f1076129ea1 _ZN2yb6Thread15SuperviseThreadEPv (libyb_util.so)
                #4  0x00007f107450a694 start_thread (libpthread.so.0)
                #5  0x00007f107424b41d __clone (libc.so.6)

GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /nfusr/dev-server/dfelsing/code/yugabyte-db/build/release-clang12-linuxbrew-dynamic-ninja/bin/yb-master...done.

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 400893]
[New LWP 369073]
[New LWP 369120]
[New LWP 369074]
[New LWP 369082]
[New LWP 369087]
[New LWP 400908]
[New LWP 369052]
[New LWP 400904]
[New LWP 369076]
[New LWP 369135]
[New LWP 374879]
[New LWP 400902]
[New LWP 400901]
[New LWP 369072]
[New LWP 434645]
[New LWP 369407]
[New LWP 369084]
[New LWP 369811]
[New LWP 400906]
[New LWP 369133]
[New LWP 400905]
[New LWP 400896]
[New LWP 369081]
[New LWP 369141]
[New LWP 400903]
[New LWP 369083]
[New LWP 369085]
[New LWP 400895]
[New LWP 369090]
[New LWP 369129]
[New LWP 369092]
[New LWP 369097]
[New LWP 400891]
[New LWP 400907]
[New LWP 369109]
[New LWP 369112]
[New LWP 369138]
[New LWP 369115]
[New LWP 369127]
[New LWP 369130]
[New LWP 369132]
[New LWP 369139]
[New LWP 369134]
[New LWP 369405]
[New LWP 369810]
[New LWP 369968]
[New LWP 371512]
[New LWP 371514]
[New LWP 372533]
[New LWP 380987]
[New LWP 400894]
[New LWP 400900]
[New LWP 380985]
[New LWP 369406]
[New LWP 369088]
[New LWP 369188]
[New LWP 380986]
[New LWP 400897]
[New LWP 371269]
[New LWP 369091]
[New LWP 369809]
[New LWP 369404]
[New LWP 371513]
[New LWP 369086]
[New LWP 369089]
[New LWP 376006]
[New LWP 460616]
[New LWP 460617]

warning: File "/opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
    add-auto-load-safe-path /opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so
line to your configuration file "/nfusr/dev-server/dfelsing/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/nfusr/dev-server/dfelsing/.gdbinit".
For more information about this security protection see the
--Type <RET> for more, q to quit, c to continue without paging--
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/nfusr/dev-server/dfelsing/code/yugabyte-db/build/latest/bin/yb-master --fs_dat'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f1074ceb7fe in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/lib/libc++.so.1
[Current thread is 1 (LWP 400893)]
(gdb) bt
#0  0x00007f1074ceb7fe in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/lib/libc++.so.1
#1  0x00007f10792ec90d in yb::master::AsyncRemoveTableFromTablet::AsyncRemoveTableFromTablet (this=0x19352698, master=<optimized out>, callback_pool=0x2a52a80, tablet=..., table=...)
    at ../../src/yb/master/async_rpc_tasks.cc:1289
#2  0x00007f10793d0ce0 in std::__1::construct_at<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, yb::master::AsyncRemoveTableFromTablet*> (__location=0x19352698, __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__memory/base.h:105
#3  std::__1::allocator_traits<std::__1::allocator<yb::master::AsyncRemoveTableFromTablet> >::construct<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void, void> (__p=0x19352698, __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__memory/allocator_traits.h:296
#4  std::__1::__shared_ptr_emplace<yb::master::AsyncRemoveTableFromTablet, std::__1::allocator<yb::master::AsyncRemoveTableFromTablet> >::__shared_ptr_emplace<yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&> (this=0x19352680, __args=..., __args=..., __args=..., __args=..., __a=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:2620
#5  std::__1::allocate_shared<yb::master::AsyncRemoveTableFromTablet, std::__1::allocator<yb::master::AsyncRemoveTableFromTablet>, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void> (__a=..., __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:3380
#6  0x00007f1079367596 in std::__1::make_shared<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void> (__args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:3389
#7  yb::master::CatalogManager::DeleteTableInternal (this=this@entry=0x2a88000, req=req@entry=0x1e4e8ad0, resp=<optimized out>, resp@entry=0x1e4e8af8, rpc=<optimized out>, rpc@entry=0x7f1057fbef48)
    at ../../src/yb/master/catalog_manager.cc:5292
#8  0x00007f10793667ff in yb::master::CatalogManager::DeleteTable (this=0x2a88000, req=0x1e4e8ad0, resp=0x1e4e8af8, rpc=0x7ffd8a174090) at ../../src/yb/master/catalog_manager.cc:5203
--Type <RET> for more, q to quit, c to continue without paging--c
#9  0x00007f1079461950 in yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}::operator()() const (this=0x7f1057fbee30) at ../../src/yb/master/master_service_base-internal.h:146
#10 yb::master::MasterServiceBase::HandleOnLeader<yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB, yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}, char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>) (this=<optimized out>, req=<optimized out>, resp=0x1e4e8af8, rpc=0x7f1057fbef48, f=..., file_name=<optimized out>, line_number=<optimized out>, line_number@entry=60, function_name=0x7f1079259f65 "DeleteTable", hold_catalog_lock=...) at ../../src/yb/master/master_service_base-internal.h:84
#11 0x00007f107945e90a in yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB> (this=0x193527b0, req=0x1c0, resp=0x15403, rpc=0x4ffbe6d6a76f, f=<optimized out>, file_name=0x7ffd8a174090 "", line_number=60, function_name=0x7a3db90 "\200\361\207\036", hold_catalog_lock=...) at ../../src/yb/master/master_service_base-internal.h:145
#12 yb::master::(anonymous namespace)::MasterDdlServiceImpl::DeleteTable (this=<optimized out>, req=0x1c0, resp=0x15403, rpc=<error reading variable: Cannot access memory at address 0x4ffbe6d6a76f>) at ../../src/yb/master/master_ddl_service.cc:29
#13 0x00007f1076dd46d4 in yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}::operator()(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext) const (this=<optimized out>, req=0x1e4e8ad0, resp=0x1e4e8af8, rpc_context=...) at src/yb/master/master_ddl.service.cc:793
#14 yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>, yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}>(std::__1::shared_ptr<yb::rpc::InboundCall>, yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}) (call=..., f=...) at ../../src/yb/rpc/local_call.h:124
#15 yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const (this=<optimized out>, call=...) at src/yb/master/master_ddl.service.cc:791
#16 std::__1::__invoke<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall> > (__f=..., __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/type_traits:3694
#17 std::__1::__invoke_void_return_wrapper<void, true>::__call<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) (__args=..., __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__functional_base:348
#18 std::__1::__function::__alloc_func<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11, std::__1::allocator<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1558
#19 std::__1::__function::__func<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11, std::__1::allocator<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1732
#20 0x00007f1076dcfca6 in std::__1::__function::__value_func<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) const (this=<optimized out>, __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#21 std::__1::function<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#22 yb::master::MasterDdlIf::Handle (this=<optimized out>, call=...) at src/yb/master/master_ddl.service.cc:611
#23 0x00007f10764d7333 in yb::rpc::ServicePoolImpl::Handle (this=0x2bb86c0, incoming=...) at ../../src/yb/rpc/service_pool.cc:270
#24 0x00007f1076480d55 in yb::rpc::InboundCall::InboundCallTask::Run (this=<optimized out>) at ../../src/yb/rpc/inbound_call.cc:221
#25 0x00007f10764e32c0 in yb::rpc::(anonymous namespace)::Worker::Execute (this=<optimized out>) at ../../src/yb/rpc/thread_pool.cc:104
#26 0x00007f1076129ea1 in std::__1::__function::__value_func<void ()>::operator()() const (this=0xcb06dd0) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#27 std::__1::function<void ()>::operator()() const (this=0xcb06dd0) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#28 yb::Thread::SuperviseThread (arg=0xcb06d80) at ../../src/yb/util/thread.cc:774
#29 0x00007f107450a694 in start_thread (arg=0x7f1057fc7700) at pthread_create.c:333
#30 0x00007f107424b41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Last master logging before crash:

I0622 10:56:02.566653 460607 backfill_index.cc:268] tg2_1 [id=00004eff000030008000000000004f73] - Alter table completed version=231, state: RUNNING
I0622 10:56:02.721055 460609 ysql_transaction_ddl.cc:55] Verifying Transaction { transaction_id: fd27f52e-2d0c-43f9-8aee-5441a77856ed isolation: SNAPSHOT_ISOLATION status_tablet: 272ec4af93c748a9856d8d7a271e1644 priority: 3257715829111520484 start_time: { physical: 1655895361911196 } locality: GLOBAL old_status_tablet: }
I0622 10:56:02.721690 380986 ysql_transaction_ddl.cc:110] Got Response for { transaction_id: fd27f52e-2d0c-43f9-8aee-5441a77856ed isolation: SNAPSHOT_ISOLATION status_tablet: 272ec4af93c748a9856d8d7a271e1644 priority: 3257715829111520484 start_time: { physical: 1655895361911196 } locality: GLOBAL old_status_tablet: }, resp: status: PENDING status_hybrid_time: 6782547405706633215 propagated_hybrid_time: 6782547405706657792 aborted_subtxn_set { }
I0622 10:56:02.938504 460609 catalog_manager.cc:3890] T 00000000000000000000000000000000 P ec795436c29b4cd7955c5a2d1f00496c: Table transaction succeeded: idx_tg2_1_c63_float8_c56_float8_c15_int_c85_date_c44_date_c2_bi [id=00004eff0000300080000000000058cb]
I0622 10:56:03.552181 400893 catalog_manager.cc:5165] Servicing DeleteTable request from 127.0.0.1:39007: table { table_id: "00004eff0000300080000000000058c3" } is_index_table: true
I0622 10:56:03.583068 400893 catalog_manager_ent.cc:2690] Deleting CDC streams for tables: 00004eff0000300080000000000058c3
I0622 10:56:03.583106 400893 catalog_manager.cc:5278] Deleting table: idx_tg3_1_c58_float4_c41_smallint_c70_numeric_c32_float8_c44_fl, retained by: []
I0622 10:56:03.583101 369139 catalog_manager.cc:5543] Marking table as DELETED: idx_tg3_1_c58_float4_c41_smallint_c70_numeric_c32_float8_c44_fl [id=00004eff0000300080000000000058c3]
I0622 10:56:03.583132 400893 catalog_manager.cc:5288] Notifying tablet with id b430bd2bc2e046739f037c5e9e35ec99 to remove this colocated table idx_tg3_1_c58_float4_c41_smallint_c70_numeric_c32_float8_c44_fl from its metadata.
I0622 10:56:03.583148 400893 catalog_entity_info.cc:821] Colocated Tablet not found for table idx_tg3_1_c58_float4_c41_smallint_c70_numeric_c32_float8_c44_fl
I0622 10:56:03.612241 460617 backfill_index.cc:268] tg3_1 [id=00004eff000030008000000000004f7f] - Alter table completed version=210, state: RUNNING

I can provide full yugabyte-data dir on request (410 MB), as well as coredump file. There is no FATAL file.

def- commented 2 years ago

So far couldn't reproduce without packed columns enabled.

def- commented 2 years ago

On state 2e8c2fce3c6af3f967bf597ac2c44b004d9cf863 with 85ac8d87d99a214d99f6e701334b3162bfedc7e5. reverted I got this again (packed columns enabled). @spolitov So it appears this is not fixed yet and an independent issue:

bash-4.4$ python3.9 ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
2022-06-24 16:09:21,218 MainThread INFO
2022-06-24 16:09:21,218 MainThread INFO     --------------------------------------------------------------------------------
2022-06-24 16:09:21,218 MainThread INFO     Running Long System Test 0.1
2022-06-24 16:09:21,218 MainThread INFO     --------------------------------------------------------------------------------
2022-06-24 16:09:21,218 MainThread INFO
2022-06-24 16:09:21,228 MainThread INFO     Reproduce with: git checkout e383187a && ./long_system_test.py --nodes=127.0.0.1:5433 --threads=10 --runtime=0 --complexity=full --max-columns=10 --seed=032216
2022-06-24 16:09:21,582 MainThread INFO     Database version: PostgreSQL 11.2-YB-2.15.1.0-b0 on x86_64-pc-linux-gnu, compiled by clang version 12.0.1 (https://github.com/yugabyte/llvm-project.git bdb147e675d8c87cee72cc1f87c4b82855977d94), 64-bit
2022-06-24 16:09:21,584 MainThread INFO     Creating tables for database db_lst_032216
2022-06-24 16:09:36,191 MainThread INFO     Starting worker_0: RandomSelectAction, SetConfigAction
2022-06-24 16:09:36,192 MainThread INFO     Starting worker_1: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-24 16:09:36,194 MainThread INFO     Starting worker_2: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-24 16:09:36,194 MainThread INFO     Starting worker_3: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-24 16:09:36,195 MainThread INFO     Starting worker_4: RandomSelectAction, SetConfigAction
2022-06-24 16:09:36,196 MainThread INFO     Starting worker_5: SingleInsertAction, SingleUpdateAction, SingleDeleteAction, BulkInsertAction, BulkUpdateAction, SetConfigAction
2022-06-24 16:09:36,196 MainThread INFO     Starting worker_6: RandomSelectAction, SetConfigAction
2022-06-24 16:09:36,197 MainThread INFO     Starting worker_7: CreateIndexAction, DropIndexAction, SetConfigAction, AddColumnAction
2022-06-24 16:09:36,198 MainThread INFO     Starting worker_8: SingleInsertAction, SingleUpdateAction, SingleDeleteAction, BulkInsertAction, BulkUpdateAction, SetConfigAction
2022-06-24 16:09:36,199 MainThread INFO     Starting worker_9: RandomSelectAction, SetConfigAction
2022-06-24 16:09:46,209 MainThread INFO     Worker queries/s: [007.7][000.3][001.0][000.4][001.4][003.4][001.2][000.6][012.8][001.3]
[...]
2022-06-24 16:57:39,222 MainThread INFO     Worker queries/s: [000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0][000.0]
Exception in thread worker_8:
Traceback (most recent call last):
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/action.py", line 91, in _exec
    cur.execute(explain_query.string)
psycopg2.errors.InternalError_: cache lookup failed for attribute -8 of relation 27105
CONTEXT:  Catalog Version Mismatch: A DDL occurred while processing this query. Try again.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/worker.py", line 65, in run
    action.run(cur)
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/action.py", line 525, in run
    self._exec(cur, Query(query, supports_explain=True, supports_rollback=True))
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/action.py", line 97, in _exec
    raise Error(explain_query, self, e)
lst.error.Error: (<lst.query.Query object at 0x7f43d4407550>, <lst.action.SingleDeleteAction object at 0x7f43d6dd0160>, InternalError_('cache lookup failed for attribute -8 of relation 27105\nCONTEXT:  Catalog Version Mismatch: A DDL occurred while processing this query. Try again.\n'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/worker.py", line 84, in run
    raise e
  File "/nfusr/dev-server/dfelsing/code/yb-long-system-test/lst/worker.py", line 76, in run
    self.conn.rollback()
psycopg2.OperationalError: Could not reconnect to database
HINT:  Database might have been dropped by another user
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

2022-06-24 16:57:47,685 worker_5   ERROR    Unexpected query failure: OperationalError
Query: EXPLAIN (ANALYZE TRUE, VERBOSE TRUE, COSTS FALSE, BUFFERS FALSE, TIMING FALSE, SUMMARY FALSE, FORMAT JSON)
INSERT INTO tg3_3 (c0_json, c1_int, c2_boolean, c3_numrange, c4_date, c5_daterange, c6_date, c7_jsonb, c8_text, c9_numrange, c10_int2range, c11_date, c12_boolean, c13_smallint, c14_numeric, c15_date, c16_daterange, c17_jsonb, c18_daterange, c19_json, c20_smallint, c21_jsonb, c22_int, c23_bigint, c24_int4range, c25_int, c26_jsonb, c27_int8range, c28_numeric, c29_float8, c30_boolean, c31_text, c32_jsonb, c33_smallint, c34_date, c35_float8, c36_boolean, c37_int4range, c38_int4range, c39_date, c40_int2range, c41_int4range, c42_numrange, c43_text, c44_json, c45_boolean, c46_jsonb, c47_boolean, c48_int8range, c49_smallint) VALUES ('{"a": 3, "b": ["0"], "c": true}'::json, -45, TRUE, '[-80.53068771612506,-27.265897591163863]'::NUMRANGE, '1988-09-02', '[1942-02-20,1983-01-22)'::DATERANGE, '1959-12-04', '{"a": 3, "b": ["0", "1", "2", "3"], "c": false}'::jsonb, '16', '(26.392787374082644,56.90547412099417)'::NUMRANGE, '(-92,-65]'::INT2RANGE, '1959-04-27', TRUE, -32768, 47.3701794113189, '1920-11-06', '(1955-09-17,1994-10-21]'::DATERANGE, '{"a": 6, "b": ["0", "1", "2", "3", "4", "5"], "c": true}'::jsonb, '(1915-12-14,1980-03-10)'::DATERANGE, '{"a": 9, "b": ["0"], "c": false}'::json, 81, '{"a": 8, "b": ["0"], "c": true}'::jsonb, -2147483648, -17, '(-18,23]'::INT4RANGE, 3, '{"a": 8, "b": ["0", "1", "2", "3", "4"], "c": true}'::jsonb, '[-93,24)'::INT8RANGE, -21.38014275537104, 54.97379265965654, TRUE, '57', '{"a": 3, "b": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"], "c": true}'::jsonb, 16, '1970-03-23', 91.34403961484855, TRUE, '[-29,55)'::INT4RANGE, '[-15,54)'::INT4RANGE, '1918-04-02', '(-74,77)'::INT2RANGE, '(-68,-37)'::INT4RANGE, '[-68.30457431933146,74.58472675660167]'::NUMRANGE, '79', '{"a": 10, "b": ["0", "1", "2", "3", "4"], "c": false}'::json, FALSE, '{"a": 9, "b": ["0", "1", "2", "3", "4", "5", "6", "7", "8"], "c": true}'::jsonb, TRUE, '[49,93]'::INT8RANGE, 100);
  values: None
  runtime: 2022-06-24 16:52:45.609 - 2022-06-24 16:57:47.684
  supports explain: False
  supports rollback: False
  affected rows: None
Action: SingleInsertAction
Error class: OperationalError
Error code: None
Error message: FATAL:  Could not reconnect to database
    @           0x997be8  errmsg
    @           0x983762  YBPreloadRelCache
    @           0x83fa65  YBPreloadRelCacheHelper
    @           0x83f88d  YBRefreshCache
    @           0x83b90d  PostgresMain
    @           0x7ad83c  BackendRun
    @           0x7acc90  ServerLoop
    @           0x7a92fb  PostmasterMain
    @           0x71302d  PostgresServerProcessMain
    @           0x7134d2  main
    @     0x7f8302a92825  __libc_start_main
    @           0x4b8a29  _start

HINT:  Database might have been dropped by another user
    @           0x9992a8  errhint
    @           0x983772  YBPreloadRelCache
    @           0x83fa65  YBPreloadRelCacheHelper
    @           0x83f88d  YBRefreshCache
    @           0x83b90d  PostgresMain
    @           0x7ad83c  BackendRun
    @           0x7acc90  ServerLoop
    @           0x7a92fb  PostmasterMain
    @           0x71302d  PostgresServerProcessMain
    @           0x7134d2  main
    @     0x7f8302a92825  __libc_start_main
    @           0x4b8a29  _start

server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
Transaction isolation level: default
DB Node: host: 127.0.0.1, port: 5433
DB Backend PID: 1938129
(gdb) bt
#0  0x00007f349f6be7fe in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/lib/libc++.so.1
#1  0x00007f34a3cf192d in yb::master::AsyncRemoveTableFromTablet::AsyncRemoveTableFromTablet (this=0x21f33058, master=<optimized out>, callback_pool=0x25bac40, tablet=..., table=...)
    at ../../src/yb/master/async_rpc_tasks.cc:1289
#2  0x00007f34a3dd62a0 in std::__1::construct_at<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, yb::master::AsyncRemoveTableFromTablet*> (__location=0x21f33058, __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__memory/base.h:105
#3  std::__1::allocator_traits<std::__1::allocator<yb::master::AsyncRemoveTableFromTablet> >::construct<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void, void> (__p=0x21f33058, __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__memory/allocator_traits.h:296
#4  std::__1::__shared_ptr_emplace<yb::master::AsyncRemoveTableFromTablet, std::__1::allocator<yb::master::AsyncRemoveTableFromTablet> >::__shared_ptr_emplace<yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&> (this=0x21f33040, __args=..., __args=..., __args=..., __args=..., __a=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:2620
#5  std::__1::allocate_shared<yb::master::AsyncRemoveTableFromTablet, std::__1::allocator<yb::master::AsyncRemoveTableFromTablet>, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void> (__a=..., __args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:3380
#6  0x00007f34a3d6cb06 in std::__1::make_shared<yb::master::AsyncRemoveTableFromTablet, yb::master::Master*&, yb::ThreadPool*, scoped_refptr<yb::master::TabletInfo>, scoped_refptr<yb::master::TableInfo> const&, void> (__args=..., __args=..., __args=..., __args=...)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/memory:3389
#7  yb::master::CatalogManager::DeleteTableInternal (this=this@entry=0x25f2000, req=req@entry=0x1074e0b0, resp=<optimized out>, resp@entry=0x1074e0d8, rpc=rpc@entry=0x7f3473172f48)
    at ../../src/yb/master/catalog_manager.cc:5296
#8  0x00007f34a3d6bd81 in yb::master::CatalogManager::DeleteTable (this=0x25f2000, req=0x1074e0b0, resp=0x1074e0d8, rpc=0x7ffc2776b090) at ../../src/yb/master/catalog_manager.cc:5207
#9  0x00007f34a3e67d50 in yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::mastr::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}::operator()() const (this=0x7f3473172e30) at ../../src/yb/master/master_service_base-internal.h:146
#10 yb::master::MasterServiceBase::HandleOnLeader<yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB, yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::{lambda()#1}, char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>) (this=<optimized out>, req=<optimized out>, resp=0x1074e0d8, rpc=0x7f3473172f48, f=..., file_name=<optimized out>, line_number=<optimized out>, line_number@entry=60, function_name=0x7f34a3c5da58 "DeleteTable", hold_catalog_lock=...) at ../../src/yb/master/master_service_base-internal.h:84
#11 0x00007f34a3e64d0a in yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB> (this=0x21f33170, req=0x1c0, resp=0x1dd0b, rpc=0x6fe3a1989952, f=<optimized out>, file_name=0x7ffc2776b090 "", line_number=60, function_name=0x115bbda0 "", hold_catalog_lock=...) at ../../src/yb/master/master_service_base-internal.h:145
#12 yb::master::(anonymous namespace)::MasterDdlServiceImpl::DeleteTable (this=<optimized out>, req=0x1c0, resp=0x1dd0b, rpc=<error reading variable: Cannot access memory at address 0x6fe3a1989952>) at ../../src/yb/master/master_ddl_service.cc:29
#13 0x00007f34a17b5894 in yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}::operator()(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext) const (this=<optimized out>, req=0x1074e0b0, resp=0x1074e0d8, rpc_context=...) at src/yb/master/master_ddl.service.cc:793
#14 yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::master::DeleteTableRequestPB, yb::master::DeleteTableResponsePB>, yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}>(std::__1::shared_ptr<yb::rpc::InboundCall>, yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::{lambda(yb::master::DeleteTableRequestPB const*, yb::master::DeleteTableResponsePB*, yb::rpc::RpcContext)#1}) (call=..., f=...) at ../../src/yb/rpc/local_call.h:124
#15 yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const (this=<optimized out>, call=...) at src/yb/master/master_ddl.service.cc:791
#16 std::__1::__invoke<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall> > (__f=..., __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/type_traits:3694
#17 std::__1::__invoke_void_return_wrapper<void, true>::__call<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) (__args=..., __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/__functional_base:348
#18 std::__1::__function::__alloc_func<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11, std::__1::allocator<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1558
#19 std::__1::__function::__func<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11, std::__1::allocator<yb::master::MasterDdlIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_11>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1732
#20 0x00007f34a17b0e66 in std::__1::__function::__value_func<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) const (this=<optimized out>, __args=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#21 std::__1::function<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#22 yb::master::MasterDdlIf::Handle (this=<optimized out>, call=...) at src/yb/master/master_ddl.service.cc:611
#23 0x00007f34a0eab44a in yb::rpc::ServicePoolImpl::Handle (this=0x27226c0, incoming=...) at ../../src/yb/rpc/service_pool.cc:270
#24 0x00007f34a0e54fc5 in yb::rpc::InboundCall::InboundCallTask::Run (this=<optimized out>) at ../../src/yb/rpc/inbound_call.cc:237
#25 0x00007f34a0eb73d0 in yb::rpc::(anonymous namespace)::Worker::Execute (this=<optimized out>) at ../../src/yb/rpc/thread_pool.cc:104
#26 0x00007f34a0afdb41 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x40f5490) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#27 std::__1::function<void ()>::operator()() const (this=0x40f5490) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181632-c1907426fc-almalinux8-x86_64-clang12-linuxbrew/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#28 yb::Thread::SuperviseThread (arg=0x40f5440) at ../../src/yb/util/thread.cc:774
#29 0x00007f349eedd694 in start_thread (arg=0x7f347317b700) at pthread_create.c:333
#30 0x00007f349ec1e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

master server stays unreachable after this.

def- commented 2 years ago

I just got this same issue with packed columns disabled (bin/yb-ctl --replication_factor 1 create --tserver_flags=yb_enable_read_committed_isolation=true --master_flags=yb_enable_read_committed_isolation=true), so it is independent of packed columns.

bmatican commented 2 years ago

@def- Any chance you can re-run with Sergei's latest fix above?

def- commented 2 years ago

Starting a test run now. Seems to take a while to crash though so might take a few days to reliably tell if it's fixed.

def- commented 2 years ago

Haven‘t seen this failure in the last 2 days, so seems to be fixed.

bmatican commented 2 years ago

Awesome, thanks!