search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.9k stars 1.06k forks source link

[DocDB] Restore after Expand/Shrink universe is failing #13372

Open Arpit-yb opened 2 years ago

Arpit-yb commented 2 years ago

Jira Link: DB-3010

Description

Steps to repro

  1. Create a universe
  2. Create Backup
  3. Add a new node
  4. Remove the node
  5. Restore

The restore task fails because a table has a LEARNER peer. The script waits for the peer removing and fails

CC: @VijiYB @bmatican

OlegLoginov commented 2 years ago

Short log:

2022-07-19 05:06:58,004 INFO: Output from running command [[ ssh ... yb-admin ... list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

...
 2022-07-19 05:16:44,690 INFO: Output from running command [[ ssh ... yb-admin ...  
list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

...
 2022-07-19 05:16:44,690 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.

The peer on TS ***94e is LEARNER for ~10 minutes. It's not an expected case.

OlegLoginov commented 2 years ago

Detailed log:

Command to run: [bin/py_wrapper bin/yb_backup.py --masters 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --ts_web_hosts_ports 10.9.212.159:9000,10.9.222.165:9000,10.9.222.238:9000 --parallelism 8 --ysql_port 5433 --keyspace postgres --no_auto_name --ssh_port 22 --ssh_key_path /opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem --ip_to_ssh_key_path {"10.9.222.238":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.212.159":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.222.165":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem"} --backup_location s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres --storage_type s3 --certs_dir /home/yugabyte/yugabyte-tls-config/ restore --verbose]
Starting proc (abbrev cmd) - bin/py_wrapper bin/yb_backup.py --masters 10.9.212.159:7100,10.9.222 ... 3 --certs_dir /home/yugabyte/yugabyte-tls-config/ restore --verbose
Starting proc (full cmd) - 'bin/py_wrapper' 'bin/yb_backup.py' '--masters' '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100' '--ts_web_hosts_ports' '10.9.212.159:9000,10.9.222.165:9000,10.9.222.238:9000' '--parallelism' '8' '--ysql_port' '5433' '--keyspace' 'postgres' '--no_auto_name' '--ssh_port' '22' '--ssh_key_path' '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' '--ip_to_ssh_key_path' '{"10.9.222.238":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.212.159":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.222.165":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem"}' '--backup_location' 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres' '--storage_type' 's3' '--certs_dir' '/home/yugabyte/yugabyte-tls-config/' 'restore' '--verbose' - logging stdout=/tmp/shell_process_out6709940775139998410tmp, stderr=/tmp/shell_process_err4326675699039409961tmp
2022-07-19 05:06:50,876 INFO: [app] YB cluster version: 2.15.1.0
2022-07-19 05:06:50,876 INFO: [app] Restoring backup from s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres
2022-07-19 05:06:53,335 INFO: [app] Loading manifest data from /tmp/yb_backup_cstgtvplygbdamug/Manifest
2022-07-19 05:06:56,943 INFO: [app] Starting phase 1: Create objects via YSQL dumps
2022-07-19 05:06:56,943 INFO: [app] Create YSQL tables from /tmp/yb_backup_cstgtvplygbdamug/YSQLDump
2022-07-19 05:06:56,991 INFO: [app] Skip renaming because YSQL DB name was not changed: 'postgres'
2022-07-19 05:06:57,595 INFO: [app] Starting phase 2: Import snapshot
2022-07-19 05:06:57,829 INFO: [app] Starting phase 3: Generate list of tservers for every tablet
2022-07-19 05:06:57,830 INFO: [app] Start searching for tablet replicas (try 0)
2022-07-19 05:07:18,069 INFO: [app] Start searching for tablet replicas (try 1)
2022-07-19 05:07:38,303 INFO: [app] Start searching for tablet replicas (try 2)
2022-07-19 05:07:58,540 INFO: [app] Start searching for tablet replicas (try 3)
2022-07-19 05:08:18,781 INFO: [app] Start searching for tablet replicas (try 4)
2022-07-19 05:08:39,003 INFO: [app] Start searching for tablet replicas (try 5)
2022-07-19 05:08:59,210 INFO: [app] Start searching for tablet replicas (try 6)
2022-07-19 05:09:19,457 INFO: [app] Start searching for tablet replicas (try 7)
2022-07-19 05:09:39,678 INFO: [app] Start searching for tablet replicas (try 8)
2022-07-19 05:09:59,885 INFO: [app] Start searching for tablet replicas (try 9)
2022-07-19 05:10:20,126 INFO: [app] Start searching for tablet replicas (try 10)
2022-07-19 05:10:40,359 INFO: [app] Start searching for tablet replicas (try 11)
2022-07-19 05:11:00,592 INFO: [app] Start searching for tablet replicas (try 12)
2022-07-19 05:11:20,833 INFO: [app] Start searching for tablet replicas (try 13)
2022-07-19 05:11:41,066 INFO: [app] Start searching for tablet replicas (try 14)
2022-07-19 05:12:01,294 INFO: [app] Start searching for tablet replicas (try 15)
2022-07-19 05:12:21,514 INFO: [app] Start searching for tablet replicas (try 16)
2022-07-19 05:12:41,739 INFO: [app] Start searching for tablet replicas (try 17)
2022-07-19 05:13:01,980 INFO: [app] Start searching for tablet replicas (try 18)
2022-07-19 05:13:22,215 INFO: [app] Start searching for tablet replicas (try 19)
2022-07-19 05:13:42,446 INFO: [app] Start searching for tablet replicas (try 20)
2022-07-19 05:14:02,677 INFO: [app] Start searching for tablet replicas (try 21)
2022-07-19 05:14:22,902 INFO: [app] Start searching for tablet replicas (try 22)
2022-07-19 05:14:43,135 INFO: [app] Start searching for tablet replicas (try 23)
2022-07-19 05:15:03,369 INFO: [app] Start searching for tablet replicas (try 24)
2022-07-19 05:15:23,595 INFO: [app] Start searching for tablet replicas (try 25)
2022-07-19 05:15:43,820 INFO: [app] Start searching for tablet replicas (try 26)
2022-07-19 05:16:04,037 INFO: [app] Start searching for tablet replicas (try 27)
2022-07-19 05:16:24,257 INFO: [app] Start searching for tablet replicas (try 28)
2022-07-19 05:16:44,479 INFO: [app] Start searching for tablet replicas (try 29)
2022-07-19 05:17:04,697 INFO: [app] Summary of run:
 Proc stdout for 'bin/py_wrapper bin/yb_backup.py --masters 10.9.212.159:7100,10.9.222 ... 3 --certs_dir /home/yugabyte/yugabyte-tls-config/ restore --verbose' :
 {"error": "Backup exception: Exceeded max number of retries for the tablet replicas searching loop (30)!"}
 [2022-07-19_05_06_49 common.sh:327 activate_virtualenv] Using virtualenv python executable now.
 [2022-07-19_05_06_49 common.sh:327 activate_virtualenv] Using virtualenv python executable now.
 2022-07-19 05:06:50,038 INFO: Parsed arguments: {'masters': '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', 'ts_web_hosts_ports': '10.9.212.159:9000,10.9.222.165:9000,10.9.222.238:9000', 'ts_secondary_ip_map': None, 'ip_to_ssh_key_path': '{"10.9.222.238":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.212.159":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem","10.9.222.165":"/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem"}', 'k8s_config': None, 'keyspace': ['postgres'], 'table': None, 'table_uuid': None, 'local_yb_admin_binary': None, 'remote_yb_admin_binary': '/home/yugabyte/master/bin/yb-admin', 'local_ysql_dump_binary': None, 'remote_ysql_dump_binary': '/home/yugabyte/master/postgres/bin/ysql_dump', 'local_ysql_shell_binary': None, 'remote_ysql_shell_binary': '/home/yugabyte/master/bin/ysqlsh', 'pg_based_backup': False, 'ssh_key_path': '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem', 'ssh_user': 'yugabyte', 'remote_user': 'yugabyte', 'ssh_port': '22', 'no_ssh': False, 'mac': False, 'ysql_port': '5433', 'ysql_host': None, 'ysql_enable_auth': False, 'disable_checksums': False, 'backup_location': 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres', 's3bucket': None, 'region': None, 'region_location': None, 'no_auto_name': True, 'no_snapshot_deleting': False, 'snapshot_id': None, 'verbose': True, 'parallelism': 8, 'disable_parallelism': False, 'storage_type': 's3', 'command': 'restore', 'certs_dir': '/home/yugabyte/yugabyte-tls-config/', 'sse': False, 'backup_keys_source': None, 'restore_keys_destination': None, 'nfs_storage_path': None, 'restore_time': None, 'use_tablespaces': False, 'upload': True, 'edit_ysql_dump_sed_reg_exp': None, 'do_not_disable_splitting': False, 'TEST_sleep_during_download_dir': False, 'TEST_sleep_after_find_snapshot_dirs': False, 'TEST_yb_admin_unsupported_commands': False, 'TEST_never_fsync': False}
 2022-07-19 05:06:50,038 INFO: Running command: mkdir -p /tmp/yb_backup_cstgtvplygbdamug
 2022-07-19 05:06:50,040 INFO: Output from running command [[ mkdir -p /tmp/yb_backup_cstgtvplygbdamug ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,040 INFO: TS Web hosts/ports: 10.9.212.159:9000,10.9.222.165:9000,10.9.222.238:9000
 2022-07-19 05:06:50,040 INFO: Uploading /tmp/yb_backup_cstgtvplygbdamug/cloud_cfg to /tmp/yb_backup_cstgtvplygbdamug on 10.9.212.159
 2022-07-19 05:06:50,040 INFO: Creating /tmp/yb_backup_cstgtvplygbdamug on server 10.9.212.159
 2022-07-19 05:06:50,040 INFO: Running command ['mkdir', '-p', '/tmp/yb_backup_cstgtvplygbdamug'] on server 10.9.212.159
 2022-07-19 05:06:50,040 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'mkdir -p /tmp/yb_backup_cstgtvplygbdamug'"'"''
 2022-07-19 05:06:50,111 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'mkdir -p /tmp/yb_backup_cstgtvplygbdamug'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,111 INFO: Running command (3 retries): scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -P 22 -q /tmp/yb_backup_cstgtvplygbdamug/cloud_cfg yugabyte@10.9.212.159:/tmp/yb_backup_cstgtvplygbdamug
 2022-07-19 05:06:50,629 INFO: Output from running command [[ scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -P 22 -q /tmp/yb_backup_cstgtvplygbdamug/cloud_cfg yugabyte@10.9.212.159:/tmp/yb_backup_cstgtvplygbdamug ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,630 INFO: Uploading /tmp/yb_backup_cstgtvplygbdamug/cloud_cfg to /tmp/yb_backup_cstgtvplygbdamug on 10.9.212.159 done: 
 2022-07-19 05:06:50,630 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_all_masters', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:50,630 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_all_masters --never_fsync=false'"'"''
 2022-07-19 05:06:50,762 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_all_masters --never_fsync=false'"'"'' ]]:
 Master UUID                        RPC Host/Port           State       Role
 60036d8a6e2b4fc6ac811f698cdb4ba8   10.9.212.159:7100       ALIVE       LEADER
 a54e1af3954d4c88a2a7c21f809c4bb4   10.9.222.165:7100       ALIVE       FOLLOWER
 4d1d0aea27dc493390fdcc37441a2fce   10.9.222.238:7100       ALIVE       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,763 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', '--version', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:50,763 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ --version --never_fsync=false'"'"''
 2022-07-19 05:06:50,876 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ --version --never_fsync=false'"'"'' ]]:
 version 2.15.1.0 build 162 revision 5c70d6b292a11b32a6609e2cab63eeaa3164b6df build_type RELEASE built at 14 Jul 2022 09:54:49 UTC

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,876 INFO: YB cluster version string: version 2.15.1.0 build 162 revision 5c70d6b292a11b32a6609e2cab63eeaa3164b6df build_type RELEASE built at 14 Jul 2022 09:54:49 UTC
 2022-07-19 05:06:50,876 INFO: [app] YB cluster version: 2.15.1.0
 2022-07-19 05:06:50,876 INFO: [app] Restoring backup from s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres
 2022-07-19 05:06:50,876 INFO: Creating /tmp/yb_backup_cstgtvplygbdamug on server 10.9.212.159
 2022-07-19 05:06:50,876 INFO: Running command ['mkdir', '-p', '/tmp/yb_backup_cstgtvplygbdamug'] on server 10.9.212.159
 2022-07-19 05:06:50,876 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'mkdir -p /tmp/yb_backup_cstgtvplygbdamug'"'"''
 2022-07-19 05:06:50,937 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'mkdir -p /tmp/yb_backup_cstgtvplygbdamug'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:50,937 INFO: Running command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'command -v /usr/bin/xxh64sum /dev/null'
 2022-07-19 05:06:51,004 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'command -v /usr/bin/xxh64sum /dev/null' ]]:
 /usr/bin/xxh64sum

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:51,004 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest.xxh64', '/tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded'] on server 10.9.212.159
 2022-07-19 05:06:51,005 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest.xxh64 /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded'"'"''
 2022-07-19 05:06:51,838 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest.xxh64 /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest.xxh64' -> '/tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded' (59 bytes in 0.3 seconds, 183.70 B/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:51,838 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest', '/tmp/yb_backup_cstgtvplygbdamug/Manifest'] on server 10.9.212.159
 2022-07-19 05:06:51,838 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest /tmp/yb_backup_cstgtvplygbdamug/Manifest'"'"''
 2022-07-19 05:06:52,703 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest /tmp/yb_backup_cstgtvplygbdamug/Manifest'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest' -> '/tmp/yb_backup_cstgtvplygbdamug/Manifest' (1291 bytes in 0.3 seconds, 3.93 kB/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:52,703 INFO: Running command /usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/Manifest > /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64 on server 10.9.212.159
 2022-07-19 05:06:52,703 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/Manifest > /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64'"'"''
 2022-07-19 05:06:52,772 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/Manifest > /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:52,772 INFO: Running command test " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded ) " = " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64 ) " && echo correct || echo invalid on server 10.9.212.159
 2022-07-19 05:06:52,772 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64 ) " && echo correct || echo invalid'"'"''
 2022-07-19 05:06:52,843 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/Manifest.xxh64 ) " && echo correct || echo invalid'"'"'' ]]:
 correct

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:52,843 INFO: Downloaded metadata file /tmp/yb_backup_cstgtvplygbdamug/Manifest from s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/Manifest
 2022-07-19 05:06:52,843 INFO: Downloading /tmp/yb_backup_cstgtvplygbdamug/Manifest to local dir /tmp/yb_backup_cstgtvplygbdamug from 10.9.212.159
 2022-07-19 05:06:52,843 INFO: Running command (3 retries): scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -P 22 -q yugabyte@10.9.212.159:/tmp/yb_backup_cstgtvplygbdamug/Manifest /tmp/yb_backup_cstgtvplygbdamug
 2022-07-19 05:06:53,335 INFO: Output from running command [[ scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -P 22 -q yugabyte@10.9.212.159:/tmp/yb_backup_cstgtvplygbdamug/Manifest /tmp/yb_backup_cstgtvplygbdamug ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:53,335 INFO: Downloading /tmp/yb_backup_cstgtvplygbdamug/Manifest to local dir /tmp/yb_backup_cstgtvplygbdamug from 10.9.212.159 done: 
 2022-07-19 05:06:53,335 INFO: [app] Loading manifest data from /tmp/yb_backup_cstgtvplygbdamug/Manifest
 2022-07-19 05:06:53,336 INFO: Loaded manifest: {
   "version": "1.0",
   "properties": {
     "storage-type": "s3",
     "platform-version": "version 2.15.1.0 build 162 revision 5c70d6b292a11b32a6609e2cab63eeaa3164b6df build_type release built at 14 Jul 2022 15:08:34 UTC",
     "parallelism": 8,
     "use-tablespaces": false,
     "pg-based-backup": false,
     "start-time": "Tue, 19 Jul 2022 04:48:18 +0000",
     "end-time": "Tue, 19 Jul 2022 04:48:35 +0000",
     "size-in-bytes": 651564,
     "check-sums": true,
     "hash-algorithm": "xxh64"
   },
   "locations": {
     "s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres": {
       "default": true,
       "tablet-directories": {
         "27132de09b724b74b2a441b19eb46c3f": {},
         "18c81765cd054e5f9e49ee36f30d3758": {},
         "15e664ec6ac74421885a0d854f0ba683": {}
       }
     }
   },
   "source": {
     "ysql": true,
     "ysql_authentication": false,
     "k8s": false,
     "yb-database-version": "version 2.15.1.0 build 162 revision 5c70d6b292a11b32a6609e2cab63eeaa3164b6df build_type RELEASE built at 14 Jul 2022 09:54:49 UTC",
     "yb-masters": [
       "10.9.212.159:7100",
       "10.9.222.165:7100",
       "10.9.222.238:7100"
     ],
     "tables": [
       "ysql.postgres.postgresqlkeyvalue"
     ]
   }
 }
 2022-07-19 05:06:53,336 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump.xxh64', '/tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded'] on server 10.9.212.159
 2022-07-19 05:06:53,336 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump.xxh64 /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded'"'"''
 2022-07-19 05:06:54,129 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump.xxh64 /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump.xxh64' -> '/tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded' (59 bytes in 0.3 seconds, 195.14 B/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:54,130 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump', '/tmp/yb_backup_cstgtvplygbdamug/YSQLDump'] on server 10.9.212.159
 2022-07-19 05:06:54,130 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump /tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"''
 2022-07-19 05:06:54,960 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump /tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump' -> '/tmp/yb_backup_cstgtvplygbdamug/YSQLDump' (1866 bytes in 0.3 seconds, 5.43 kB/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:54,960 INFO: Running command /usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/YSQLDump > /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64 on server 10.9.212.159
 2022-07-19 05:06:54,960 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/YSQLDump > /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64'"'"''
 2022-07-19 05:06:55,008 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/YSQLDump > /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:55,008 INFO: Running command test " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded ) " = " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64 ) " && echo correct || echo invalid on server 10.9.212.159
 2022-07-19 05:06:55,008 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64 ) " && echo correct || echo invalid'"'"''
 2022-07-19 05:06:55,058 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump.xxh64 ) " && echo correct || echo invalid'"'"'' ]]:
 correct

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:55,058 INFO: Downloaded metadata file /tmp/yb_backup_cstgtvplygbdamug/YSQLDump from s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/YSQLDump
 2022-07-19 05:06:55,058 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB.xxh64', '/tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded'] on server 10.9.212.159
 2022-07-19 05:06:55,058 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB.xxh64 /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded'"'"''
 2022-07-19 05:06:55,936 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB.xxh64 /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB.xxh64' -> '/tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded' (65 bytes in 0.4 seconds, 178.60 B/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:55,936 INFO: Running command ['s3cmd', '--force', '--no-check-certificate', '--config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg', 'get', 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB', '/tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB'] on server 10.9.212.159
 2022-07-19 05:06:55,936 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB'"'"''
 2022-07-19 05:06:56,844 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'s3cmd --force --no-check-certificate --config=/tmp/yb_backup_cstgtvplygbdamug/cloud_cfg get s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB'"'"'' ]]:
 download: 's3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB' -> '/tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB' (28788 bytes in 0.4 seconds, 69.63 kB/s)

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:56,844 INFO: Running command /usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB > /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64 on server 10.9.212.159
 2022-07-19 05:06:56,844 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB > /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64'"'"''
 2022-07-19 05:06:56,892 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/usr/bin/xxh64sum /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB > /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:56,893 INFO: Running command test " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded ) " = " $( sed 's| .*/| |' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64 ) " && echo correct || echo invalid on server 10.9.212.159
 2022-07-19 05:06:56,893 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64 ) " && echo correct || echo invalid'"'"''
 2022-07-19 05:06:56,943 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'test " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64.downloaded ) " = " $( sed '"'"'"'"'"'"'"'"'s| .*/| |'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB.xxh64 ) " && echo correct || echo invalid'"'"'' ]]:
 correct

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:56,943 INFO: Downloaded metadata file /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB from s3://backups.yugabyte.com/test/itest/univ-a71fb09b-39fa-47ff-be78-02d6e8c23f84/backup-2022-07-19T04:48:17-251874474/keyspace-postgres/SnapshotInfoPB
 2022-07-19 05:06:56,943 INFO: Completed phase 0: START [Time taken for phase: 0:00:06.908286]
 2022-07-19 05:06:56,943 INFO: [app] Starting phase 1: Create objects via YSQL dumps
 2022-07-19 05:06:56,943 INFO: [app] Create YSQL tables from /tmp/yb_backup_cstgtvplygbdamug/YSQLDump
 2022-07-19 05:06:56,943 INFO: Running command sed -n '/CREATE DATABASE/{s|CREATE DATABASE||;s|WITH.*||;p}' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump on server 10.9.212.159
 2022-07-19 05:06:56,943 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'sed -n '"'"'"'"'"'"'"'"'/CREATE DATABASE/{s|CREATE DATABASE||;s|WITH.*||;p}'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"''
 2022-07-19 05:06:56,991 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'sed -n '"'"'"'"'"'"'"'"'/CREATE DATABASE/{s|CREATE DATABASE||;s|WITH.*||;p}'"'"'"'"'"'"'"'"' /tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"'' ]]:
  postgres 

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:56,991 INFO: [app] Skip renaming because YSQL DB name was not changed: 'postgres'
 2022-07-19 05:06:56,992 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_all_tablet_servers', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:56,992 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_all_tablet_servers --never_fsync=false'"'"''
 2022-07-19 05:06:57,104 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_all_tablet_servers --never_fsync=false'"'"'' ]]:
 Tablet Server UUID               RPC Host/Port Heartbeat delay Status   Reads/s  Writes/s Uptime   SST total size  SST uncomp size SST #files      Memory   
 e1670c2d3f0b45a2833dcccf2e23f867 10.9.222.238:9100 0.18s           ALIVE    0.00     0.00     167      9.54 MB         30.20 MB        31              99.81 MB 
 50b845918e704518bce5257e829d52ce 10.9.208.4:9100 1733.75s        DEAD     0.00     0.00     0        0 B             0 B             0               0 B      
 92ffdf44ea9044ce9cf89deb00519883 10.9.211.91:9100 1733.75s        DEAD     0.00     0.00     0        0 B             0 B             0               0 B      
 b48a054a12c74de28a75ea0e312c89b9 10.9.222.165:9100 0.18s           ALIVE    0.00     0.00     259      8.40 MB         24.95 MB        31              106.86 MB 
 069c5ac91e1c4ba2ad1ded67b349ac09 10.9.212.159:9100 0.18s           ALIVE    0.00     0.00     213      8.86 MB         25.72 MB        35              123.81 MB 
 1a4db47f50b047b0bc490497a3a8d94e 10.9.214.176:9100 1733.75s        DEAD     0.00     0.00     0        0 B             0 B             0               0 B      

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:57,105 INFO: Selecting alive TS 10.9.222.165 from ['10.9.222.238', '10.9.222.165', '10.9.212.159']
 2022-07-19 05:06:57,105 INFO: Running command ['/home/yugabyte/master/bin/ysqlsh', '--host=10.9.222.165', '--port=5433', '--dbname=template1', '--echo-all', '--file=/tmp/yb_backup_cstgtvplygbdamug/YSQLDump'] on server 10.9.212.159
 2022-07-19 05:06:57,105 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/ysqlsh --host=10.9.222.165 --port=5433 --dbname=template1 --echo-all --file=/tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"''
 2022-07-19 05:06:57,595 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/ysqlsh --host=10.9.222.165 --port=5433 --dbname=template1 --echo-all --file=/tmp/yb_backup_cstgtvplygbdamug/YSQLDump'"'"'' ]]:
 --
 -- YSQL database dump
 --
 -- Dumped from database version 11.2-YB-2.15.1.0-b0
 -- Dumped by ysql_dump version 11.2-YB-2.15.1.0-b0
 SET yb_binary_restore = true;
 SET
 SET statement_timeout = 0;
 SET
 SET lock_timeout = 0;
 SET
 SET idle_in_transaction_session_timeout = 0;
 SET
 SET client_encoding = 'UTF8';
 SET
 SET standard_conforming_strings = on;
 SET
 SELECT pg_catalog.set_config('search_path', '', false);
  set_config 
 ------------

 (1 row)

 SET check_function_bodies = false;
 SET
 SET client_min_messages = warning;
 SET
 SET row_security = off;
 SET
 --
 -- Name: postgres; Type: DATABASE; Schema: -; Owner: postgres
 --
 CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'C' LC_CTYPE = 'en_US.UTF-8';
 ALTER DATABASE postgres OWNER TO postgres;
 ALTER DATABASE
 \connect postgres
 ysqlsh:/tmp/yb_backup_cstgtvplygbdamug/YSQLDump:23: ERROR:  database "postgres" already exists
 You are now connected to database "postgres" as user "yugabyte".
 SET yb_binary_restore = true;
 SET
 SET statement_timeout = 0;
 SET
 SET lock_timeout = 0;
 SET
 SET idle_in_transaction_session_timeout = 0;
 SET
 SET client_encoding = 'UTF8';
 SET
 SET standard_conforming_strings = on;
 SET
 SELECT pg_catalog.set_config('search_path', '', false);
  set_config 
 ------------

 (1 row)

 SET check_function_bodies = false;
 SET
 SET client_min_messages = warning;
 SET
 SET row_security = off;
 SET
 --
 -- Name: DATABASE postgres; Type: COMMENT; Schema: -; Owner: postgres
 --
 COMMENT ON DATABASE postgres IS 'default administrative connection database';
 COMMENT
 SET default_with_oids = false;
 SET
 --
 -- Name: postgresqlkeyvalue; Type: TABLE; Schema: public; Owner: postgres
 --
 -- For binary upgrade, must preserve pg_type oid
 SELECT pg_catalog.binary_upgrade_set_next_pg_type_oid('16391'::pg_catalog.oid);
  binary_upgrade_set_next_pg_type_oid 
 -------------------------------------

 (1 row)

 -- For binary upgrade, must preserve pg_type array oid
 SELECT pg_catalog.binary_upgrade_set_next_array_pg_type_oid('16390'::pg_catalog.oid);
  binary_upgrade_set_next_array_pg_type_oid 
 -------------------------------------------

 (1 row)

 CREATE TABLE public.postgresqlkeyvalue (
     k text NOT NULL,
     v text,
     CONSTRAINT postgresqlkeyvalue_pkey PRIMARY KEY((k) HASH)
 )
 SPLIT INTO 3 TABLETS;
 ALTER TABLE public.postgresqlkeyvalue OWNER TO postgres;
 ysqlsh:/tmp/yb_backup_cstgtvplygbdamug/YSQLDump:67: ERROR:  relation "postgresqlkeyvalue" already exists
 ALTER TABLE
 --
 -- YSQL database dump complete
 --

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:57,595 INFO: Completed phase 1: Create objects via YSQL dumps [Time taken for phase: 0:00:00.652018]
 2022-07-19 05:06:57,595 INFO: [app] Starting phase 2: Import snapshot
 2022-07-19 05:06:57,595 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'import_snapshot', '/tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB', 'postgres', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:57,595 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ import_snapshot /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB postgres --never_fsync=false'"'"''
 2022-07-19 05:06:57,714 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ import_snapshot /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB postgres --never_fsync=false'"'"'' ]]:
 Read snapshot meta file /tmp/yb_backup_cstgtvplygbdamug/SnapshotInfoPB
 Importing snapshot 1bf204a1-afa2-46de-b85e-c4e7dcbc5828 (COMPLETE)
 Table type: table
 Target imported table name: postgres.postgresqlkeyvalue
 Table being imported: postgres.postgresqlkeyvalue
 Successfully applied snapshot.
 Object             Old ID                              New ID                          
 Keyspace           000033e6000030008000000000000000    000033e6000030008000000000000000
 Table              000033e6000030008000000000004005    000033e6000030008000000000004005
 Tablet 0           27132de09b724b74b2a441b19eb46c3f    27132de09b724b74b2a441b19eb46c3f
 Tablet 1           18c81765cd054e5f9e49ee36f30d3758    18c81765cd054e5f9e49ee36f30d3758
 Tablet 2           15e664ec6ac74421885a0d854f0ba683    15e664ec6ac74421885a0d854f0ba683
 Snapshot           1bf204a1-afa2-46de-b85e-c4e7dcbc5828    49f00fda-a980-457b-a183-502615134200

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:57,714 INFO: Imported table: postgres.postgresqlkeyvalue
 2022-07-19 05:06:57,714 INFO: Imported table id was changed from 000033e6000030008000000000004005 to 000033e6000030008000000000004005
 2022-07-19 05:06:57,715 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_snapshots', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:57,715 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_snapshots --never_fsync=false'"'"''
 2022-07-19 05:06:57,829 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_snapshots --never_fsync=false'"'"'' ]]:
 Snapshot UUID                      State       Creation Time
 49f00fda-a980-457b-a183-502615134200   COMPLETE    2022-07-19 05:06:58.199047
 Restoration UUID                   State
 4e1ac0a9-5b27-45a6-8a28-fe21c4b391db   RESTORED

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:57,829 INFO: Snapshot id 49f00fda-a980-457b-a183-502615134200 importing completed successfully
 2022-07-19 05:06:57,829 INFO: Snapshot 49f00fda-a980-457b-a183-502615134200 will be deleted at exit...
 2022-07-19 05:06:57,829 INFO: Completed phase 2: Import snapshot [Time taken for phase: 0:00:00.234084]
 2022-07-19 05:06:57,829 INFO: [app] Starting phase 3: Generate list of tservers for every tablet
 2022-07-19 05:06:57,830 INFO: [app] Start searching for tablet replicas (try 0)
 2022-07-19 05:06:57,831 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:57,831 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:57,831 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:06:57,831 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:06:57,831 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:06:57,831 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''

 2022-07-19 05:06:58,004 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ 
list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:58,031 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:58,049 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:06:58,049 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:06:58,049 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:07:18,069 INFO: [app] Start searching for tablet replicas (try 1)
 2022-07-19 05:07:18,069 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:18,069 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:18,070 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:18,070 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:07:18,070 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:07:18,070 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:07:18,277 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ 
list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:18,278 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:18,283 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:18,283 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:07:18,283 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:07:38,303 INFO: [app] Start searching for tablet replicas (try 2)
 2022-07-19 05:07:38,304 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:38,304 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:38,304 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:38,304 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:07:38,304 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:07:38,304 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:07:38,508 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:38,510 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ 
list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:38,520 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:38,520 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:07:38,520 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:07:58,540 INFO: [app] Start searching for tablet replicas (try 3)
 2022-07-19 05:07:58,541 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:58,541 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:58,541 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:07:58,541 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:07:58,541 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:07:58,541 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:07:58,742 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:58,768 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:58,776 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:07:58,776 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:07:58,776 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:08:18,781 INFO: [app] Start searching for tablet replicas (try 4)
 2022-07-19 05:08:18,782 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:18,782 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:08:18,782 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:18,782 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:18,782 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:08:18,784 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:08:18,939 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:18,987 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:18,991 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:18,991 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:08:18,991 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:08:39,003 INFO: [app] Start searching for tablet replicas (try 5)
 2022-07-19 05:08:39,004 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:39,004 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:39,004 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:39,004 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:08:39,004 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:08:39,004 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:08:39,159 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:39,198 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:39,209 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:39,210 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:08:39,210 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:08:59,210 INFO: [app] Start searching for tablet replicas (try 6)
 2022-07-19 05:08:59,211 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:59,211 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:59,211 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:08:59,211 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:08:59,211 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:08:59,211 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:08:59,386 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:59,437 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:59,437 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:08:59,438 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:08:59,438 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:09:19,457 INFO: [app] Start searching for tablet replicas (try 7)
 2022-07-19 05:09:19,457 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:19,457 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:19,458 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:19,458 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:09:19,458 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:09:19,458 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:09:19,618 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:19,669 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:19,672 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:19,673 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:09:19,673 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:09:39,678 INFO: [app] Start searching for tablet replicas (try 8)
 2022-07-19 05:09:39,679 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:39,679 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:39,679 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:39,679 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:09:39,679 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:09:39,679 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:09:39,879 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:39,884 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:39,884 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:09:39,885 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:09:39,885 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:09:59,885 INFO: [app] Start searching for tablet replicas (try 9)
 2022-07-19 05:09:59,886 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:59,886 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:59,886 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:09:59,886 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:09:59,886 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:09:59,886 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:10:00,100 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:00,100 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:00,106 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:00,106 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:10:00,106 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:10:20,126 INFO: [app] Start searching for tablet replicas (try 10)
 2022-07-19 05:10:20,127 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:20,127 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:20,127 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:20,127 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:10:20,127 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:10:20,127 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:10:20,280 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:20,320 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:20,339 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:20,340 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:10:20,340 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:10:40,359 INFO: [app] Start searching for tablet replicas (try 11)
 2022-07-19 05:10:40,359 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:40,360 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:10:40,360 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:40,360 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:10:40,360 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:10:40,360 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:10:40,516 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:40,568 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:40,572 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:10:40,572 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:10:40,572 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:11:00,592 INFO: [app] Start searching for tablet replicas (try 12)
 2022-07-19 05:11:00,592 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:00,593 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:00,593 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:00,593 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:11:00,593 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:11:00,593 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:11:00,806 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:00,806 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:00,817 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:00,817 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:11:00,817 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:11:20,833 INFO: [app] Start searching for tablet replicas (try 13)
 2022-07-19 05:11:20,833 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:20,833 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:20,833 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:20,833 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:11:20,834 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:11:20,834 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:11:21,033 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:21,042 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:21,046 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:21,046 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:11:21,046 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:11:41,066 INFO: [app] Start searching for tablet replicas (try 14)
 2022-07-19 05:11:41,067 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:41,067 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:41,067 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:11:41,067 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:11:41,067 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:11:41,067 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:11:41,245 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:41,268 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:41,278 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:11:41,278 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:11:41,278 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:12:01,294 INFO: [app] Start searching for tablet replicas (try 15)
 2022-07-19 05:12:01,294 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:01,294 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:01,294 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:01,294 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:12:01,294 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:12:01,295 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:12:01,485 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:01,501 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:01,512 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:01,512 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:12:01,512 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:12:21,514 INFO: [app] Start searching for tablet replicas (try 16)
 2022-07-19 05:12:21,514 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:21,514 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:21,514 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:21,514 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:12:21,515 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:12:21,515 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:12:21,705 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:21,713 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:21,719 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:21,719 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:12:21,719 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:12:41,739 INFO: [app] Start searching for tablet replicas (try 17)
 2022-07-19 05:12:41,739 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:41,739 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:41,740 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:12:41,740 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:12:41,740 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:12:41,740 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:12:41,909 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:41,938 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:41,960 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:12:41,960 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:12:41,960 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:13:01,980 INFO: [app] Start searching for tablet replicas (try 18)
 2022-07-19 05:13:01,981 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:01,981 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:01,981 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:01,981 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:13:01,981 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:13:01,981 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:13:02,161 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:02,190 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:02,194 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:02,195 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:13:02,195 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:13:22,215 INFO: [app] Start searching for tablet replicas (try 19)
 2022-07-19 05:13:22,215 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:22,216 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:22,216 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:22,216 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:13:22,216 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:13:22,216 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:13:22,406 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:22,421 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:22,426 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:22,426 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:13:22,426 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:13:42,446 INFO: [app] Start searching for tablet replicas (try 20)
 2022-07-19 05:13:42,446 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:42,446 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:42,447 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:13:42,447 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:13:42,447 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:13:42,447 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:13:42,622 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:42,623 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:42,670 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:13:42,670 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:13:42,670 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:14:02,677 INFO: [app] Start searching for tablet replicas (try 21)
 2022-07-19 05:14:02,678 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:02,678 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:02,678 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:02,678 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:14:02,678 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:14:02,678 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:14:02,884 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:02,900 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:02,900 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:02,901 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:14:02,901 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:14:22,902 INFO: [app] Start searching for tablet replicas (try 22)
 2022-07-19 05:14:22,902 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:22,902 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:22,903 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:22,903 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:14:22,903 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:14:22,903 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:14:23,118 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:23,128 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:23,133 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:23,133 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:14:23,133 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:14:43,135 INFO: [app] Start searching for tablet replicas (try 23)
 2022-07-19 05:14:43,136 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:43,136 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:43,136 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:14:43,136 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:14:43,136 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:14:43,136 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:14:43,331 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:43,343 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:43,351 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:14:43,352 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:14:43,352 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:15:03,369 INFO: [app] Start searching for tablet replicas (try 24)
 2022-07-19 05:15:03,369 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:03,370 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:03,370 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:03,370 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:15:03,370 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:15:03,370 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:15:03,564 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:03,580 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:03,584 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:03,584 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:15:03,584 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:15:23,595 INFO: [app] Start searching for tablet replicas (try 25)
 2022-07-19 05:15:23,595 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:23,596 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:23,596 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:23,596 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:15:23,596 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:15:23,596 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:15:23,789 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:23,796 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:23,799 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:23,799 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:15:23,800 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:15:43,820 INFO: [app] Start searching for tablet replicas (try 26)
 2022-07-19 05:15:43,820 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:43,820 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:43,820 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:15:43,820 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:15:43,820 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:15:43,821 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:15:43,974 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:44,022 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:44,031 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:15:44,032 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:15:44,032 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:16:04,037 INFO: [app] Start searching for tablet replicas (try 27)
 2022-07-19 05:16:04,038 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:04,038 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:04,038 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:16:04,038 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:16:04,038 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:04,041 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:16:04,199 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:04,249 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:04,257 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:04,257 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:16:04,257 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:16:24,257 INFO: [app] Start searching for tablet replicas (try 28)
 2022-07-19 05:16:24,258 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:24,258 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:24,258 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:24,258 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:16:24,258 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:16:24,258 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:16:24,456 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:24,461 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:24,466 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:24,466 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:16:24,466 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:16:44,479 INFO: [app] Start searching for tablet replicas (try 29)
 2022-07-19 05:16:44,480 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '15e664ec6ac74421885a0d854f0ba683', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:44,480 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '18c81765cd054e5f9e49ee36f30d3758', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:44,480 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'list_tablet_servers', '27132de09b724b74b2a441b19eb46c3f', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:16:44,480 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"''
 2022-07-19 05:16:44,480 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"''
 2022-07-19 05:16:44,480 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"''
 2022-07-19 05:16:44,686 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 27132de09b724b74b2a441b19eb46c3f --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:44,687 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 15e664ec6ac74421885a0d854f0ba683 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       LEADER
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER

 [[ END OF OUTPUT ]]

 2022-07-19 05:16:44,690 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ list_tablet_servers 
18c81765cd054e5f9e49ee36f30d3758 --never_fsync=false'"'"'' ]]:
 Server UUID                        RPC Host/Port           Role
 b48a054a12c74de28a75ea0e312c89b9   10.9.222.165:9100       LEADER
 069c5ac91e1c4ba2ad1ded67b349ac09   10.9.212.159:9100       FOLLOWER
 e1670c2d3f0b45a2833dcccf2e23f867   10.9.222.238:9100       FOLLOWER
 1a4db47f50b047b0bc490497a3a8d94e   10.9.214.176:9100       LEARNER

 [[ END OF OUTPUT ]]
 2022-07-19 05:16:44,690 WARNING: Found TS 10.9.214.176 with bad role: LEARNER for tablet 18c81765cd054e5f9e49ee36f30d3758. Retry searching.
 2022-07-19 05:16:44,690 INFO: Sleep for 20 seconds before the next tablet replicas searching round.
 2022-07-19 05:17:04,697 INFO: Completed phase 3: Generate list of tservers for every tablet [Time taken for phase: 0:10:06.867728]
 2022-07-19 05:17:04,697 INFO: [app] Summary of run:
 0:00:00.652018 : PHASE 1 : Create objects via YSQL dumps
 0:00:00.234084 : PHASE 2 : Import snapshot
 0:10:06.867728 : PHASE 3 : Generate list of tservers for every tablet
 Total runtime: 0:10:14.662116
 2022-07-19 05:17:04,697 INFO: Terminating all threadpool ...
 2022-07-19 05:17:04,697 INFO: Terminating threadpool ...
 2022-07-19 05:17:04,755 INFO: Terminated threadpool ...
 2022-07-19 05:17:04,855 INFO: Deleting snapshot 49f00fda-a980-457b-a183-502615134200 ...
 2022-07-19 05:17:04,855 INFO: Running command ['/home/yugabyte/master/bin/yb-admin', '--master_addresses', '10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100', '--certs_dir_name', '/home/yugabyte/yugabyte-tls-config/', 'delete_snapshot', '49f00fda-a980-457b-a183-502615134200', '--never_fsync=false'] on server 10.9.212.159
 2022-07-19 05:17:04,855 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ delete_snapshot 49f00fda-a980-457b-a183-502615134200 --never_fsync=false'"'"''
 2022-07-19 05:17:04,984 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'/home/yugabyte/master/bin/yb-admin --master_addresses 10.9.212.159:7100,10.9.222.165:7100,10.9.222.238:7100 --certs_dir_name /home/yugabyte/yugabyte-tls-config/ delete_snapshot 49f00fda-a980-457b-a183-502615134200 --never_fsync=false'"'"'' ]]:
 Deleted snapshot: 49f00fda-a980-457b-a183-502615134200

 [[ END OF OUTPUT ]]
 2022-07-19 05:17:04,985 INFO: Removing remote temporary directory '/tmp/yb_backup_cstgtvplygbdamug' on 10.9.212.159
 2022-07-19 05:17:04,985 INFO: Running command ['rm', '-rf', '/tmp/yb_backup_cstgtvplygbdamug'] on server 10.9.212.159
 2022-07-19 05:17:04,985 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'rm -rf /tmp/yb_backup_cstgtvplygbdamug'"'"''
 2022-07-19 05:17:05,085 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'rm -rf /tmp/yb_backup_cstgtvplygbdamug'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:17:05,085 INFO: Removing remote temporary directory '/tmp/yb_backup_cstgtvplygbdamug' on 10.9.212.159
 2022-07-19 05:17:05,085 INFO: Running command ['rm', '-rf', '/tmp/yb_backup_cstgtvplygbdamug'] on server 10.9.212.159
 2022-07-19 05:17:05,085 INFO: Running command (10 retries): ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'rm -rf /tmp/yb_backup_cstgtvplygbdamug'"'"''
 2022-07-19 05:17:05,244 INFO: Output from running command [[ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o 'ControlPath=~/.ssh/ssh-%r@%h:%p' -o ControlPersist=1m -i '/opt/yugabyte/yugaware/data/keys/eaf4484c-6d99-4326-98f7-563b418b7a61/onprem-ONPREM NTON TLS-itest-key-20220719-043433.pem' -p 22 -q yugabyte@10.9.212.159 'cd / &&  bash -c '"'"'rm -rf /tmp/yb_backup_cstgtvplygbdamug'"'"'' ]]:

 [[ END OF OUTPUT ]]
 2022-07-19 05:17:05,244 INFO: Removing temporary directory '/tmp/yb_backup_cstgtvplygbdamug'
 2022-07-19 05:17:05,244 INFO: Running command: rm -rf /tmp/yb_backup_cstgtvplygbdamug
 2022-07-19 05:17:05,246 INFO: Output from running command [[ rm -rf /tmp/yb_backup_cstgtvplygbdamug ]]:

 [[ END OF OUTPUT ]]
 Completed proc 'bin/py_wrapper bin/yb_backup.py --masters 10.9.212.159:7100,10.9.222 ... 3 --certs_dir /home/yugabyte/yugabyte-tls-config/ restore --verbose' status=success [ 615452 ms ]
 Response code=0, hasError=true.
 Errored out with: java.lang.RuntimeException: {"error": "Backup exception: Exceeded max number of retries for the tablet replicas searching loop (30)!"}
OlegLoginov commented 2 years ago

Slack thread: https://yugabyte.slack.com/archives/C01CB38CZHU/p1656999767720239

OlegLoginov commented 2 years ago

Currently it's not quite clear why we have the LEANER for 10 minutes. Possibly it's a DocDB bug. Investigating.

rthallamko3 commented 2 years ago

@Huqicheng , Can you take a look at the raft failures? If you need more context on PITR, ping @sanketkedia , @spolitov

Huqicheng commented 2 years ago 
W0719 05:17:02.595964 21016 consensus_peers.cc:543] T 79ca98e835854fb28277cc0ec104db0e P b48a054a12c74de28a75ea0e312c89b9 -> Peer 1a4db47f50b047b0bc490497a3a8d94e ([host: "10.9.214.176" port: 9100], []): Couldn't send request.  Status: Network error (yb/util/net/socket.cc:540): recvmsg error: Connection refused (system error 111). Retrying in the next heartbeat period. Already tried 1668 times. State: 2

@Arpit-yb Does "remove a node" means shutdown a tserver? Seems the server cannot be connected from leader.

Huqicheng commented 2 years ago

LB added load to a dead tserver because a tserver is set to dead only if not receiving its heartbeat for 60s. So there is a gap between resetting the blacklist and removing the tserver. LB still treat the tserver as alive and try to move loads to it.

Is it reasonable to set tserver as dead on master as the last step of node removal? @bmatican

Huqicheng commented 2 years ago

We can wait for tablet servers gone before returning shrink is successfully done.

As @rahuldesirazu suggested, we can let shrink do the following steps: Blacklist the nodes Destroy the instances Wait for the tservers to be marked as dead with ListTabletServers (this will take 90s to finish) Clear the blacklist

@OlegLoginov which one will you prefer?

  1. wait for 20 mins until the dead node is kicked out, seems obvious and straightforward.
  2. we make sure tservers are marked as dead at master side before returning success for shrink?
rthallamko3 commented 2 years ago

@lingamsandeep , This seems to be on the control path. Assigning to you to take it forward.