search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.91k stars 1.06k forks source link

Assert `metric_.get() == active_metric()` after tserver restart with new ip but same hostname. #461

Closed bbaddepudi closed 6 years ago

bbaddepudi commented 6 years ago

Testing locally on Mac. Added the following at the end of /etc/hosts ` 127.0.0.1 node1

127.0.0.2 node2

127.0.0.3 node3 `

Then started 3 masters on yb-master --webserver_interface 127.0.0.1 --rpc_bind_addresses=127.0.0.1 --server_broadcast_addresses=node1:7100 --master_addresses node1:7100,node2:7100,node3:7100 --fs_data_dirs "/tmp/yblocal1/" >& /tmp/yb-master_1.out & on 127.0.0.2/3 ips also.

Then 3 tservers yb-tserver --webserver_interface 127.0.0.1 --rpc_bind_addresses=127.0.0.1 --server_broadcast_addresses=node1:9100 --tserver_master_addrs node1:7100,node2:7100,node3:7100 --fs_data_dirs "/tmp/yblocal1/" >& /tmp/yb-tserver_1.out & on 127.0.0.2/3 ips also.

Killed the master-3 and tserver-3 processes. Changed last line in /etc/hosts to 127.0.0.4 node3

Then restarted both master-3 and tserver-3 with new ip in rpc/web addresses. master startup was fine. tserver ran into this DCHECK: F20180904 17:15:49 ../../../../../src/yb/util/net/dns_resolver.cc:90] Check failed: metric_.get() == active_metric() (0x10aede9c0 vs. 0x10a5e8d20) @ 0x10826ef55 google::LogMessage::SendToLog() @ 0x10826f981 google::LogMessage::Flush() @ 0x10827354f google::LogMessageFatal::~LogMessageFatal() @ 0x1082701f9 google::LogMessageFatal::~LogMessageFatal() @ 0x107b4097d yb::ScopedDnsTracker::~ScopedDnsTracker() @ 0x107b409f5 yb::ScopedDnsTracker::~ScopedDnsTracker() @ 0x1045339f5 yb::consensus::RaftConsensus::RefreshConsensusQueueAndPeersUnlocked() @ 0x104529349 yb::consensus::RaftConsensus::BecomeLeaderUnlocked() @ 0x104550682 yb::consensus::RaftConsensus::DoElectionCallback() @ 0x10457fbe5 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRNS_6__bindIMN2yb9consensus13RaftConsensusEFvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERKNS5_14ElectionResultEEJNS_10shared_ptrIS6_EESE_SH_EEEEEEvDpOT_ @ 0x10457f599 _ZNSt3__110__function6__funcINS_6__bindIMN2yb9consensus13RaftConsensusEFvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERKNS4_14ElectionResultEEJNS_10shared_ptrIS5_EESD_SG_EEENS9_ISL_EEFvvEEclEv @ 0x107b70e7e std::__1::function<>::operator()() @ 0x107c13269 yb::FunctionRunnable::Run() @ 0x107c0c548 yb::ThreadPool::DispatchThread() @ 0x107c24ea5 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRNS_6__bindIRKMN2yb10ThreadPoolEFvbEJRKPS5_RKbEEEEEEvDpOT_ @ 0x107c24cd9 _ZNSt3__110__function6__funcINS_6__bindIRKMN2yb10ThreadPoolEFvbEJRKPS4_RKbEEENS_9allocatorISE_EEFvvEEclEv @ 0x107b70e7e std::__1::function<>::operator()() @ 0x107bf531f yb::Thread::SuperviseThread() @ 0x7fff8676899d _pthread_body @ 0x7fff8676891a _pthread_start @ 0x7fff86766351 thread_start

bbaddepudi commented 6 years ago

Did a rebuild with thridparty and issue did not appear


09:23 $ ps -ef | grep yb-mas
  501 11359  3946   0  9:20AM ttys011    0:01.38 /Users/bharat/code/yugabyte/build/latest/bin/yb-master --webserver_interface 127.0.0.1 --rpc_bind_addresses=127.0.0.1 --server_broadcast_addresses=node1:7100 --master_addresses node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal1/
  501 11386  3946   0  9:20AM ttys011    0:01.73 /Users/bharat/code/yugabyte/build/latest/bin/yb-master --webserver_interface 127.0.0.2 --rpc_bind_addresses=127.0.0.2 --server_broadcast_addresses=node2:7100 --master_addresses node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal2/
  501 11676  3946   0  9:21AM ttys011    0:00.61 /Users/bharat/code/yugabyte/build/latest/bin/yb-master --webserver_interface 127.0.0.4 --rpc_bind_addresses=127.0.0.4 --server_broadcast_addresses=node3:7100 --master_addresses node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal3/
  501 11808  3946   0  9:23AM ttys011    0:00.00 grep yb-mas
Bharats-MacBook-Pro.local:~ 
09:23 $ ps -ef | grep yb-tser
  501 11480  3946   0  9:20AM ttys011    0:08.30 /Users/bharat/code/yugabyte/build/latest/bin/yb-tserver --webserver_interface 127.0.0.1 --rpc_bind_addresses=127.0.0.1 --server_broadcast_addresses=node1:9100 --tserver_master_addrs node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal1/
  501 11507  3946   0  9:20AM ttys011    0:08.62 /Users/bharat/code/yugabyte/build/latest/bin/yb-tserver --webserver_interface 127.0.0.2 --rpc_bind_addresses=127.0.0.2 --server_broadcast_addresses=node2:9100 --tserver_master_addrs node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal2/
  501 11705  3946   0  9:21AM ttys011    0:01.46 /Users/bharat/code/yugabyte/build/latest/bin/yb-tserver --webserver_interface 127.0.0.4 --rpc_bind_addresses=127.0.0.4 --server_broadcast_addresses=node3:9100 --tserver_master_addrs node1:7100,node2:7100,node3:7100 --fs_data_dirs /tmp/yblocal3/