vvkgopalan
commented
4 years ago This issue still exists - ran into it when creating a single node cluster with server process verbosity level of 3. --v=4 also produces an error.
$ ./bin/yb-ctl destroy
Destroying cluster.
$ ./bin/yb-ctl create --v=3
Creating cluster.
Waiting for cluster to be ready.
Traceback (most recent call last):
File "/Users/vivekgopalan/code/yugabyte-db/submodules/yugabyte-installation/bin/yb-ctl", line 2021, in
Cluster creation is fine for --v=2 or --v=1. Same goes for cluster w/ rf of 3.
11:35 $ bin/yb-ctl destroy Destroying cluster. Sankeths-MacBook-Pro.local:~/code/yugabyte-db [yb_test_config_change ↑·1|✔] 11:41 $ bin/yb-ctl --rf=3 create --v=3 Creating cluster. Waiting for cluster to be ready. Viewing file /Users/sanketh/yugabyte-data/node-2/disk-1/master.err: F0909 11:41:07.678411 64503808 operation.h:190] Check failed: hybridtime.is_valid() Fatal failure details written to /Users/sanketh/yugabyte-data/node-2/disk-1/yb-data/master/logs/yb-master.FATAL.details.2019-09-09T11_41_07.pid2154.txt F20190909 11:41:07 ../../src/yb/tablet/operations/operation.h:190] Check failed: hybridtime.is_valid() @ 0x109ca8380 google::LogDestination::LogToSinks() @ 0x109ca73de google::LogMessage::SendToLog() @ 0x109ca7d55 google::LogMessage::Flush() @ 0x109cac80f google::LogMessageFatal::~LogMessageFatal() @ 0x109ca8d29 google::LogMessageFatal::~LogMessageFatal() @ 0x102c0b56b yb::tablet::OperationState::hybrid_time() @ 0x102cfd3cf yb::tablet::SnapshotOperationState::ToString() @ 0x102cfdf8b yb::tablet::SnapshotOperation::ToString() @ 0x102cdd839 yb::tablet::OperationDriver::ToStringUnlocked() @ 0x102cdd7b9 yb::tablet::OperationDriver::ToString() @ 0x102d30384 yb::tablet::PreparerImpl::ReplicateSubBatch() @ 0x102d2ff00 yb::tablet::PreparerImpl::ProcessAndClearLeaderSideBatch() @ 0x102d2f9cb yb::tablet::PreparerImpl::Run() @ 0x10a5a7df8 yb::ThreadPool::DispatchThread() @ 0x10a594e84 yb::Thread::SuperviseThread() @ 0x7fff7e11c2eb _pthread_body @ 0x7fff7e11f249 _pthread_start @ 0x7fff7e11b40d thread_start
Check failure stack trace: @ 0x109ca7683 google::LogMessage::SendToLog() @ 0x109ca7d55 google::LogMessage::Flush() @ 0x109cac80f google::LogMessageFatal::~LogMessageFatal() @ 0x109ca8d29 google::LogMessageFatal::~LogMessageFatal() @ 0x102c0b56b yb::tablet::OperationState::hybrid_time() @ 0x102cfd3cf yb::tablet::SnapshotOperationState::ToString() @ 0x102cfdf8b yb::tablet::SnapshotOperation::ToString() @ 0x102cdd839 yb::tablet::OperationDriver::ToStringUnlocked() @ 0x102cdd7b9 yb::tablet::OperationDriver::ToString() @ 0x102d30384 yb::tablet::PreparerImpl::ReplicateSubBatch() @ 0x102d2ff00 yb::tablet::PreparerImpl::ProcessAndClearLeaderSideBatch() @ 0x102d2f9cb yb::tablet::PreparerImpl::Run() @ 0x10a5a7df8 yb::ThreadPool::DispatchThread() @ 0x10a594e84 yb::Thread::SuperviseThread() @ 0x7fff7e11c2eb _pthread_body @ 0x7fff7e11f249 _pthread_start @ 0x7fff7e11b40d thread_start Viewing file /var/folders/ss/grx0j3sx4jdf3_s9cfx5kr6h0000gn/T/tmpMTwPnr: 2019-09-09 11:41:07,031 INFO: Starting master-1 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-master --fs_data_dirs "/Users/sanketh/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --replication_factor=3 --yb_num_shards_per_tserver 2 --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --use_initial_sys_catalog_snapshot >"/Users/sanketh/yugabyte-data/node-1/disk-1/master.out" 2>"/Users/sanketh/yugabyte-data/node-1/disk-1/master.err" & 2019-09-09 11:41:07,092 INFO: Starting master-2 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-master --fs_data_dirs "/Users/sanketh/yugabyte-data/node-2/disk-1" --webserver_interface 127.0.0.2 --rpc_bind_addresses 127.0.0.2 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --replication_factor=3 --yb_num_shards_per_tserver 2 --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --use_initial_sys_catalog_snapshot >"/Users/sanketh/yugabyte-data/node-2/disk-1/master.out" 2>"/Users/sanketh/yugabyte-data/node-2/disk-1/master.err" & 2019-09-09 11:41:07,155 INFO: Starting master-3 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-master --fs_data_dirs "/Users/sanketh/yugabyte-data/node-3/disk-1" --webserver_interface 127.0.0.3 --rpc_bind_addresses 127.0.0.3 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --replication_factor=3 --yb_num_shards_per_tserver 2 --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --use_initial_sys_catalog_snapshot >"/Users/sanketh/yugabyte-data/node-3/disk-1/master.out" 2>"/Users/sanketh/yugabyte-data/node-3/disk-1/master.err" & 2019-09-09 11:41:07,214 INFO: Starting tserver-1 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-tserver --fs_data_dirs "/Users/sanketh/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --tserver_master_addrs=127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.1:6379 --cql_proxy_bind_address=127.0.0.1:9042 --local_ip_for_outbound_sockets=127.0.0.1 --use_cassandra_authentication=false --start_pgsql_proxy --pgsql_proxy_bind_address=127.0.0.1:5433 >"/Users/sanketh/yugabyte-data/node-1/disk-1/tserver.out" 2>"/Users/sanketh/yugabyte-data/node-1/disk-1/tserver.err" & 2019-09-09 11:41:07,277 INFO: Starting tserver-2 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-tserver --fs_data_dirs "/Users/sanketh/yugabyte-data/node-2/disk-1" --webserver_interface 127.0.0.2 --rpc_bind_addresses 127.0.0.2 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --tserver_master_addrs=127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.2:6379 --cql_proxy_bind_address=127.0.0.2:9042 --local_ip_for_outbound_sockets=127.0.0.2 --use_cassandra_authentication=false --start_pgsql_proxy --pgsql_proxy_bind_address=127.0.0.2:5433 >"/Users/sanketh/yugabyte-data/node-2/disk-1/tserver.out" 2>"/Users/sanketh/yugabyte-data/node-2/disk-1/tserver.err" & 2019-09-09 11:41:07,337 INFO: Starting tserver-3 with: /Users/sanketh/code/yugabyte-db/build/latest/bin/yb-tserver --fs_data_dirs "/Users/sanketh/yugabyte-data/node-3/disk-1" --webserver_interface 127.0.0.3 --rpc_bind_addresses 127.0.0.3 --v 3 --version_file_json_path=/Users/sanketh/code/yugabyte-db/build/debug-clang-dynamic-ninja --callhome_enabled=false --tserver_master_addrs=127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.3:6379 --cql_proxy_bind_address=127.0.0.3:9042 --local_ip_for_outbound_sockets=127.0.0.3 --use_cassandra_authentication=false --start_pgsql_proxy --pgsql_proxy_bind_address=127.0.0.3:5433 >"/Users/sanketh/yugabyte-data/node-3/disk-1/tserver.out" 2>"/Users/sanketh/yugabyte-data/node-3/disk-1/tserver.err" & 2019-09-09 11:41:07,343 INFO: Waiting for master and tserver processes to come up. 2019-09-09 11:41:07,673 INFO: Waiting for master leader election and tablet server registration. 2019-09-09 11:41:08,399 INFO: PIDs found: {'tserver': [2160, 2163, 2168], 'master': [2151, None, 2157]} 2019-09-09 11:41:08,399 ERROR: At least one master or tserver process is down. ^^^ Encountered errors ^^^