search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.99k stars 1.07k forks source link

[YSQL] YBCDestroyPgGate called twice in PgMiniTest.WriteRetry #3236

Closed mbautin closed 4 years ago

mbautin commented 4 years ago 
[ RUN      ] PgMiniTest.WriteRetry
The files belonging to this database system will be owned by user "mbautin".
This user must also own the server process.

In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/1557The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /private/tmp/ybtest-501/pg_mini-test.PgMiniTest.WriteRetry.1577768074092276-19178/minicluster-data/ts-0-root/pg_data ... ok
creating subdirectories ... ok
selecting default max_connections ... 300
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
syncing data to disk ... ok
2019-12-30 20:54:58.357 PST [19200] LOG:  YugaByte is ENABLED in PostgreSQL. Transactions are enabled.
2019-12-30 20:54:58.359 PST [19200] LOG:  listening on IPv4 address "127.0.0.2", port 27495
2019-12-30 20:54:58.373 PST [19201] LOG:  database system was shut down at 2019-12-30 20:54:58 PST
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1230 20:54:58.373962 206869952 webserver.cc:148] Starting webserver on 127.0.0.2:13000
I1230 20:54:58.374320 206869952 webserver.cc:153] Document root: /Users/mbautin/code/yugabyte-db5/www
I1230 20:54:58.375098 206869952 webserver.cc:240] Webserver started. Bound to: http://127.0.0.2:13000/
2019-12-30 20:54:58.378 PST [19200] LOG:  database system is ready to accept connections
I1230 20:54:58.438290 206869952 pggate.cc:105] Reset YSQL bind address to 0.0.0.0:5432
I1230 20:54:58.438943 206869952 mem_tracker.cc:249] MemTracker: hard memory limit is 8.000000 GB
I1230 20:54:58.439106 206869952 mem_tracker.cc:251] MemTracker: soft memory limit is 6.800000 GB
W1230 20:54:58.441483 206869952 hybrid_clock.cc:115] HybridClock initialized in local mode (OS X only). Not suitable for distributed clusters.
I1230 20:54:58.441751 37277696 async_initializer.cc:73] Starting to init ybclient
W1230 20:55:03.451520 37277696 net_util.cc:369] Time spent looking up canonical hostname for localhost mbmbp.local: real 5.004s user 0.000s sys 0.001s
W1230 20:55:03.451915 37277696 client-internal.cc:830] Failed to get hostname: Network error (yb/util/net/net_util.cc:376): Unable to lookup FQDN (mbmbp.local), getaddrinfo returned 8 (EAI_NONAME): Undefined error: 0 (system error 0)
I1230 20:55:03.457404 37277696 async_initializer.cc:77] Successfully built ybclient
I1230 20:55:05.269498 206869952 ybccmds.c:472] Creating Table postgres.public.t
I1230 20:55:05.269651 206869952 pg_ddl.cc:183] PgCreateTable: creating a transactional table: postgres.t
I1230 20:55:05.566455 206869952 table_creator.cc:270] Created table postgres.t of type PGSQL_TABLE_TYPE
I1230 20:55:05.992771 38350848 tablet_rpc.cc:343] Already present (yb/consensus/retryable_requests.cc:239): Failed Write(tablet: 995687a3326f4fbbb981cd08c2eeff0a, num_ops: 1, num_attempts: 5, txn: 00000000-0000-0000-0000-000000000000) to tablet 995687a3326f4fbbb981cd08c2eeff0a on tablet server { uuid: d8df31b1f8c4423e8f244bb9badd688d private: [host: "127.0.0.4" port: 22468] public: [host: "127.0.0.5.ip.yugabyte" port: 22468] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" after 5 attempt(s): Duplicate request
2019-12-30 20:55:06.004 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.004 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (1)
2019-12-30 20:55:06.062 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.062 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (12)
2019-12-30 20:55:06.092 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.092 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (13)
2019-12-30 20:55:06.130 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.130 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (17)
2019-12-30 20:55:06.161 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.161 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (19)
2019-12-30 20:55:06.229 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.229 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (21)
2019-12-30 20:55:06.274 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.274 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (27)
2019-12-30 20:55:06.480 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.480 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (28)
2019-12-30 20:55:06.507 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.507 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (30)
2019-12-30 20:55:06.526 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.526 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (31)
2019-12-30 20:55:06.549 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.549 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (32)
2019-12-30 20:55:06.582 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.582 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (34)
2019-12-30 20:55:06.647 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.647 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (37)
2019-12-30 20:55:06.787 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.787 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (43)
2019-12-30 20:55:06.811 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.811 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (44)
2019-12-30 20:55:06.848 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.848 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (49)
2019-12-30 20:55:06.899 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.899 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (58)
2019-12-30 20:55:06.938 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.938 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (62)
I1230 20:55:07.009588 37814272 tablet_rpc.cc:343] Already present (yb/consensus/retryable_requests.cc:239): Failed Write(tablet: 995687a3326f4fbbb981cd08c2eeff0a, num_ops: 1, num_attempts: 9, txn: 00000000-0000-0000-0000-000000000000) to tablet 995687a3326f4fbbb981cd08c2eeff0a on tablet server { uuid: d8df31b1f8c4423e8f244bb9badd688d private: [host: "127.0.0.4" port: 22468] public: [host: "127.0.0.5.ip.yugabyte" port: 22468] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" after 9 attempt(s): Duplicate request
2019-12-30 20:55:07.010 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.010 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (64)
2019-12-30 20:55:07.038 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.038 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (66)
2019-12-30 20:55:07.061 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.061 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (70)
2019-12-30 20:55:07.094 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.094 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (72)
2019-12-30 20:55:07.145 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.145 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (82)
2019-12-30 20:55:07.212 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.212 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (83)
2019-12-30 20:55:07.234 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.234 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (84)
2019-12-30 20:55:07.282 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.282 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (93)
2019-12-30 20:55:07.364 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.364 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (1)
2019-12-30 20:55:07.364 PST [19200] LOG:  received immediate shutdown request
2019-12-30 20:55:07.365 PST [19205] WARNING:  terminating connection because of crash of another server process
2019-12-30 20:55:07.365 PST [19205] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-12-30 20:55:07.365 PST [19205] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
F1230 20:55:07.365284 206869952 ybc_pggate.cc:76] void yb::pggate::YBCDestroyPgGate() can only be called once
Fatal failure details written to /.FATAL.details.2019-12-30T20_55_07.pid19205.txt
F20191230 20:55:07 ../../src/yb/yql/pggate/ybc_pggate.cc:76] void yb::pggate::YBCDestroyPgGate() can only be called once
    @        0x10e7b09d0  google::LogDestination::LogToSinks()
    @        0x10e7afa2e  google::LogMessage::SendToLog()
    @        0x10e7b03a5  google::LogMessage::Flush()
    @        0x10e7b527f  google::LogMessageFatal::~LogMessageFatal()
    @        0x10e7b1379  google::LogMessageFatal::~LogMessageFatal()
    @        0x10762867e  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106dbdcbb  quickdie
    @     0x7fff6e844b5d  _sigtramp
    @                0x0  (unknown)
    @        0x10b4da71c  yb::ConditionVariable::TimedWait()
    @        0x10b4da6c6  yb::ConditionVariable::WaitUntil()
    @        0x10b4db25a  yb::CountDownLatch::WaitUntil()
    @        0x10b4db2c9  yb::CountDownLatch::WaitFor()
    @        0x10b68e404  yb::ThreadJoiner::Join()
    @        0x109fd8d56  yb::rpc::Reactor::Join()
    @        0x109f94990  yb::rpc::Messenger::Shutdown()
    @        0x10762f2c2  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762f41e  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762858d  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106d99034  proc_exit
    @        0x106dbf7f6  PostgresMain
    @        0x106d434fa  BackendRun
    @        0x106d42d80  BackendStartup
    @        0x106d42332  ServerLoop
    @        0x106d40185  PostmasterMain
    @        0x106caac2e  PostgresServerProcessMain
    @        0x106caaf72  main
    @     0x7fff6e6593d5  start
    @               0x13  (unknown)

*** Check failure stack trace: ***
    @        0x10e7afcd3  google::LogMessage::SendToLog()
    @        0x10e7b03a5  google::LogMessage::Flush()
    @        0x10e7b527f  google::LogMessageFatal::~LogMessageFatal()
    @        0x10e7b1379  google::LogMessageFatal::~LogMessageFatal()
    @        0x10762867e  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106dbdcbb  quickdie
    @     0x7fff6e844b5d  _sigtramp
    @                0x0  (unknown)
    @        0x10b4da71c  yb::ConditionVariable::TimedWait()
    @        0x10b4da6c6  yb::ConditionVariable::WaitUntil()
    @        0x10b4db25a  yb::CountDownLatch::WaitUntil()
    @        0x10b4db2c9  yb::CountDownLatch::WaitFor()
    @        0x10b68e404  yb::ThreadJoiner::Join()
    @        0x109fd8d56  yb::rpc::Reactor::Join()
    @        0x109f94990  yb::rpc::Messenger::Shutdown()
    @        0x10762f2c2  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762f41e  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762858d  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106d99034  proc_exit
    @        0x106dbf7f6  PostgresMain
    @        0x106d434fa  BackendRun
    @        0x106d42d80  BackendStartup
    @        0x106d42332  ServerLoop
    @        0x106d40185  PostmasterMain
    @        0x106caac2e  PostgresServerProcessMain
    @        0x106caaf72  main
    @     0x7fff6e6593d5  start
    @               0x13  (unknown)
2019-12-30 20:55:07.377 PST [19200] LOG:  database system is shut down
bmatican commented 4 years ago

Looking over our 600 builds, just for FATAL errors, seems like we're hitting this quite a lot:

cat fatals_600 | xargs grep -l 'void yb::pggate::YBCDestroyPgGate()' | wc -l
70

Example tests:

/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/607/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/607/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/624/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/626/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/629/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/634/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/635/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/638/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/639/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/640/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/642/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/643/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/645/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/646/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/648/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/651/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/651/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/654/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/655/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/656/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/656/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/658/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/659/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/660/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/661/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/667/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/670/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/675/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/676/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/677/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/683/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/683/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/684/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/686/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/686/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/687/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/688/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/689/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/696/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/603/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/605/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/615/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBMarkDeleted.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/617/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/635/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/647/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/648/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/651/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/655/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/661/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/666/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/628/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/643/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/657/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/665/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/602/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/607/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/614/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/624/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/625/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/633/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/637/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/639/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/645/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/658/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/622/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/664/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/669/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/672/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/678/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/681/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log

cc @d-uspenskiy

This also generates various use-after-free errors in ASAN:

    #0 0x14c2215 in GetMemoryChunkContext
    #1 0x14c4a91 in pfree
    #2 0x143b976 in errhint
    #3 0x1099f28 in quickdie
    #4 0x7fb2455fbb9f  (
    #5 0x7fb2455f800c in __pthread_cond_wait
    #6 0x7fb23f497189 in void boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&)
    #7 0x7fb23f49656e in boost::asio::detail::conditionally_enabled_event::wait(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&)
    #8 0x7fb23f494c8d in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)
    #9 0x7fb23f494885 in boost::asio::detail::scheduler::run(boost::system::error_code&)
    #10 0x7fb23f48cf1a in yb::rpc::IoThreadPool::Impl::Execute()
#0 0x7f2fd02b4033 in std::__1::unique_ptr<yb::client::YBClient::Data, std::__1::default_delete<yb::client::YBClient::Data> >::operator->() const
#1 0x7f2fd02b4033 in yb::client::YBClient::IsMultiMaster() const
#2 0x7f2fd031f4e4 in yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::GetYsqlCatalogConfigRequestPB, yb::master::GetYsqlCatalogConfigResponsePB>(std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::d
#3 0x7f2fd02a1e53 in yb::client::YBClient::GetYsqlCatalogMasterVersion(unsigned long*)
#4 0x7f2fd2a1a9f5 in yb::pggate::PgSession::GetCatalogMasterVersion(unsigned long*)
#5 0x7f2fd29fd49c in yb::pggate::PgApiImpl::GetCatalogMasterVersion(unsigned long*)
#6 0x7f2fd29e98d7 in YBCPgGetCatalogMasterVersion
#7 0x10a3105 in YBRefreshCache
#8 0x10a3fec in YBCheckSharedCatalogCacheVersion
#9 0x109cfbf in PostgresMain
#10 0xef516b in BackendRun
#11 0xef4074 in BackendStartup
#12 0xef24c0 in ServerLoop
#13 0xeee4fa in PostmasterMain
#14 0xcd8113 in PostgresServerProcessMain
#15 0xcd87b1 in main (
#0 0x7f39ab8e25a5 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (
#1 0x7f39bcf3e109 in yb::HostPort::HostPort(yb::HostPort const&)
#2 0x7f39bcf65847 in yb::client::YBClient::Data::leader_master_hostport() const
#3 0x7f39bcf9c435 in yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::GetYsqlCatalogConfigRequestPB, yb::master::GetYsqlCatalogConfigResponsePB>(std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::d
#4 0x7f39bcf1ee53 in yb::client::YBClient::GetYsqlCatalogMasterVersion(unsigned long*)
#5 0x7f39bf6979f5 in yb::pggate::PgSession::GetCatalogMasterVersion(unsigned long*)
#6 0x7f39bf67a49c in yb::pggate::PgApiImpl::GetCatalogMasterVersion(unsigned long*)
#7 0x7f39bf6668d7 in YBCPgGetCatalogMasterVersion
#8 0x10a3105 in YBRefreshCache
#9 0x10a3fec in YBCheckSharedCatalogCacheVersion
#10 0x109cfbf in PostgresMain
#11 0xef516b in BackendRun
#12 0xef4074 in BackendStartup
#13 0xef24c0 in ServerLoop
#14 0xeee4fa in PostmasterMain
#15 0xcd8113 in PostgresServerProcessMain
#16 0xcd87b1 in main (
mbautin commented 4 years ago

@d-uspenskiy: can this issue be closed now?