Peer refresh failed, error: Doing another configuration change

GoedendagMC commented 2 years ago

typesense-server[3701]: E20220408 20:40:41.351882 3863 raft_server.h:61] Peer refresh failed, error: Doing another configuration change

when running systemctl status typesense-server.service

Visiting port 8108 in browser works, port 8107 doesn't.

Do I have to specify some special things in the config or something? I have no idea what this means.

Typesense 0.23.0.rc20

GoedendagMC commented 2 years ago

Log file created at: 2022/04/08 20:11:05
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:11:05.355340   937 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:11:05.355579   937 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:11:05.355870   937 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:11:05.359163   937 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:11:05.416023   937 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:11:05.464010  1068 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:11:05.465003   937 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:11:05.466487   937 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:11:05.467510  1069 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:11:05.504230  1069 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:11:05.507614  1068 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:11:05.507658  1068 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:11:05.507958  1068 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:11:05.508002  1068 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:11:05.508419  1068 store.h:283] rm /var/lib/typesense/db success
I20220408 20:11:05.508464  1068 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:11:05.529855  1068 store.h:307] DB open success!
I20220408 20:11:05.529872  1068 raft_server.cpp:469] Loading collections from disk...
I20220408 20:11:05.529884  1068 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:11:05.529928  1068 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:11:05.529933  1068 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:11:05.529937  1068 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:11:05.530675  1068 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:11:05.531091  1068 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:11:05.531111  1068 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:11:05.531545  1068 log.cpp:674] Use crc32c as the checksum type of appending entries
W20220408 20:11:05.531586  1068 protobuf_file.cpp:91] open file failed, path: /var/lib/typesense/state/log/log_meta: FILE_ERROR_NOT_FOUND
W20220408 20:11:05.531592  1068 log.cpp:685] /var/lib/typesense/state/log is empty
I20220408 20:11:05.550757  1068 log.cpp:1136] log save_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 19111
W20220408 20:11:05.550935  1068 protobuf_file.cpp:91] open file failed, path: /var/lib/typesense/state/meta/raft_meta: FILE_ERROR_NOT_FOUND
I20220408 20:11:05.550952  1068 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 1 votedfor 0.0.0.0:0:0 time: 20
W20220408 20:11:05.551077  1068 protobuf_file.cpp:91] open file failed, path: /var/lib/typesense/state/meta/raft_meta: FILE_ERROR_NOT_FOUND
I20220408 20:11:05.551084  1068 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 1 votedfor 0.0.0.0:0:0 time: 8
I20220408 20:11:05.551090  1068 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 1 last_log_id: (index=0,term=0) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:11:05.551123  1068 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 1 start vote and grant vote self
I20220408 20:11:05.555815  1068 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 2 votedfor 127.0.0.1:8107:8108 time: 4667
I20220408 20:11:05.555893  1068 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 2 become leader of group 127.0.0.1:8107:8108 
I20220408 20:11:05.557384  1075 log.cpp:108] Created new segment `/var/lib/typesense/state/log/log_inprogress_00000000000000000001' with fd=24
I20220408 20:11:05.557569  1075 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:11:05.557595  1075 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:11:05.557605  1075 raft_server.h:234] Node becomes leader, term: 2
I20220408 20:11:05.557629  1068 raft_server.cpp:127] Node last_index: 1
I20220408 20:11:05.557636  1068 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:11:05.557641  1068 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:11:05.557654  1068 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:11:05.557672  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:05.557865  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:11:15.560397  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:15.560469  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:11:25.563536  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:25.563824  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:11:35.567478  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:35.567792  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:11:45.568965  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:45.569056  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:11:55.571290  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:11:55.571401  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:05.579465  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:05.579697  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:06.536250  1069 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:12:15.581377  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:15.581593  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:25.586009  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:25.586278  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:35.589069  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:35.589273  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:45.596617  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:45.596742  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:12:55.598408  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:12:55.598537  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:05.600440  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:05.600589  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:07.549688  1069 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:13:15.601920  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:15.602015  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:25.608819  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:25.609081  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:35.611891  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:35.612180  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:45.613476  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:45.613626  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:13:55.616482  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:13:55.616675  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:05.617774  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:05.618072  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:08.563745  1069 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:14:15.619655  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:15.619796  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:25.622761  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:25.623180  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:35.625345  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:35.625443  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:45.626974  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:45.632398  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:14:55.629048  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:14:55.629148  1077 raft_server.h:59] Peer refresh succeeded!
I20220408 20:15:05.631140  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:15:05.631584  1072 raft_server.h:59] Peer refresh succeeded!
I20220408 20:15:09.584471  1069 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:15:15.633916  1068 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 1, known_applied_index: 1, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:15:15.634245  1075 raft_server.h:59] Peer refresh succeeded!
I20220408 20:15:24.869729   937 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:15:25.637392  1068 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:15:25.637755  1068 raft_server.cpp:716] Set shutting_down = true
I20220408 20:15:25.637768  1068 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:15:25.637773  1068 raft_server.cpp:726] Replication state shutdown, store sequence: 0
I20220408 20:15:25.637794  1068 raft_server.cpp:730] node->shutdown
I20220408 20:15:25.637811  1068 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 2 state LEADER
I20220408 20:15:25.638037  1068 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:15:25.638182  1068 raft_server.cpp:734] node->join
I20220408 20:15:25.638257  1077 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:15:25.638373  1077 raft_server.h:243] This node is down
I20220408 20:15:25.638442  1068 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 2 state SHUTDOWN
I20220408 20:15:25.638777  1068 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:15:25.638793  1068 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:15:25.639379  1068 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:15:25.639498  1068 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:15:25.640761  1068 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:15:25.640785  1068 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:15:26.602727  1069 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:15:26.618784  1069 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:15:26.622923  1068 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:15:26.625464  1068 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:15:26.628635   937 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:15:26.628687   937 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:15:26.628772   937 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:15:26.628805   937 typesense_server_utils.cpp:459] Deleting server
I20220408 20:15:26.628917   937 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:15:26.631659   937 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:18:11
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:18:11.746502   773 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:18:11.746699   773 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:18:11.747160   773 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:18:11.751641   773 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:18:11.779273   773 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:18:11.812747   773 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:18:11.812901   904 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:18:11.813077   773 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:18:11.813228   905 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:18:11.814453   905 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:18:11.830606   904 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:18:11.830651   904 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:18:11.831712   904 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:18:11.832875   904 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:18:11.833717   904 store.h:283] rm /var/lib/typesense/db success
I20220408 20:18:11.833775   904 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:18:11.854890   904 store.h:307] DB open success!
I20220408 20:18:11.854912   904 raft_server.cpp:469] Loading collections from disk...
I20220408 20:18:11.855182   904 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:18:11.855432   904 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:18:11.855444   904 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:18:11.855449   904 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:18:11.856098   904 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:18:11.856953   904 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:18:11.856973   904 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:18:11.857455   904 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:18:11.858428   904 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 959
I20220408 20:18:11.858498   904 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:18:11.859203   904 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 2 votedfor 127.0.0.1:8107:8108 time: 152
I20220408 20:18:11.859262   904 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 2 votedfor 127.0.0.1:8107:8108 time: 5
I20220408 20:18:11.859270   904 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 2 last_log_id: (index=1,term=2) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:18:11.859308   904 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 2 start vote and grant vote self
I20220408 20:18:11.861048   904 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 3 votedfor 127.0.0.1:8107:8108 time: 835
I20220408 20:18:11.861064   904 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 3 become leader of group 127.0.0.1:8107:8108 
I20220408 20:18:11.861136   904 raft_server.cpp:127] Node last_index: 1
I20220408 20:18:11.861145   904 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:18:11.861150   904 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:18:11.861188   904 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:18:11.861202   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 0
W20220408 20:18:11.861212   904 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:18:11.861236   904 raft_server.cpp:578] Node not ready yet (known_applied_index is 0).
E20220408 20:18:11.861348   942 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:18:11.861922   945 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:18:11.861936   945 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:18:11.861950   945 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:18:11.861958   945 raft_server.h:234] Node becomes leader, term: 3
I20220408 20:18:21.869267   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:18:21.870023   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:18:31.871233   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:18:31.871541   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:18:41.873670   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:18:41.873839   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:18:51.875478   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:18:51.875612   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:01.879302   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:01.879477   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:11.881670   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:11.881958   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:12.854712   905 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:19:21.883457   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:21.883569   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:31.889463   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:31.889860   939 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:41.899478   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:41.900969   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:19:51.901139   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:19:51.901258   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:01.911409   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:01.911557   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:11.922201   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:11.922590   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:13.882717   905 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:20:21.931593   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:21.932314   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:31.935552   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:31.935678   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:41.937093   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:41.937227   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:20:51.939285   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:20:51.939571   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:01.942660   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:01.942878   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:11.945875   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:11.946298   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:14.912959   905 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:21:21.950280   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:21.950624   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:31.953843   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:31.954679   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:41.955368   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:41.955489   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:21:51.958721   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:21:51.958889   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:01.960559   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:01.960798   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:11.962301   904 raft_server.cpp:524] Term: 3, last_index index: 2, committed_index: 2, known_applied_index: 2, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:11.962474   947 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:15.869151   773 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:22:15.932233   905 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:22:15.963024   904 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:22:15.963086   904 raft_server.cpp:716] Set shutting_down = true
I20220408 20:22:15.963091   904 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:22:15.963095   904 raft_server.cpp:726] Replication state shutdown, store sequence: 0
I20220408 20:22:15.963101   904 raft_server.cpp:730] node->shutdown
I20220408 20:22:15.963122   904 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 3 state LEADER
I20220408 20:22:15.963310   904 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:22:15.963356   904 raft_server.cpp:734] node->join
I20220408 20:22:15.963400   947 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:22:15.963480   947 raft_server.h:243] This node is down
I20220408 20:22:15.963575   904 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 3 state SHUTDOWN
I20220408 20:22:15.963820   904 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:22:15.963835   904 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:22:15.964102   904 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:22:15.964251   904 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:22:15.965317   904 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:22:15.965332   904 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:22:16.932451   905 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:22:16.932755   905 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:22:16.934526   904 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:22:16.940480   904 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:22:16.941701   773 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:22:16.941851   773 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:22:16.941929   773 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:22:16.941956   773 typesense_server_utils.cpp:459] Deleting server
I20220408 20:22:16.942060   773 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:22:16.942757   773 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:22:16
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:22:16.968219  1106 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:22:16.968314  1106 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:22:16.968554  1106 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:22:16.971952  1106 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:22:16.989681  1106 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:22:17.006894  1106 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:22:17.007154  1106 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:22:17.007294  1235 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:22:17.007969  1236 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:22:17.011114  1236 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:22:17.019387  1235 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:22:17.019424  1235 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:22:17.019637  1235 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:22:17.019733  1235 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:22:17.020339  1235 store.h:283] rm /var/lib/typesense/db success
I20220408 20:22:17.020392  1235 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:22:17.030458  1235 store.h:307] DB open success!
I20220408 20:22:17.030499  1235 raft_server.cpp:469] Loading collections from disk...
I20220408 20:22:17.030529  1235 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:22:17.030581  1235 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:22:17.030587  1235 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:22:17.030606  1235 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:22:17.031198  1235 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:22:17.031827  1235 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:22:17.031894  1235 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:22:17.032209  1235 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:22:17.032289  1235 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 71
I20220408 20:22:17.032331  1235 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:22:17.032444  1235 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 3 votedfor 127.0.0.1:8107:8108 time: 18
I20220408 20:22:17.032507  1235 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 3 votedfor 127.0.0.1:8107:8108 time: 5
I20220408 20:22:17.032516  1235 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 3 last_log_id: (index=2,term=3) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:22:17.032539  1235 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 3 start vote and grant vote self
I20220408 20:22:17.034641  1235 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 4 votedfor 127.0.0.1:8107:8108 time: 1599
I20220408 20:22:17.034667  1235 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 4 become leader of group 127.0.0.1:8107:8108 
I20220408 20:22:17.034713  1235 raft_server.cpp:127] Node last_index: 2
I20220408 20:22:17.034719  1235 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:22:17.034723  1235 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:22:17.034729  1235 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:22:17.034741  1235 raft_server.cpp:524] Term: 4, last_index index: 2, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 0
W20220408 20:22:17.034749  1235 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:22:17.034770  1235 raft_server.cpp:578] Node not ready yet (known_applied_index is 0).
E20220408 20:22:17.035181  1262 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:22:17.041122  1261 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:22:17.041170  1261 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:22:17.041175  1261 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:22:17.041189  1261 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:22:17.041208  1261 raft_server.h:234] Node becomes leader, term: 4
I20220408 20:22:27.038003  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:27.038322  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:37.040190  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:37.040462  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:47.041836  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:47.041968  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:22:57.043902  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:22:57.044054  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:07.050897  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:07.051262  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:17.052942  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:17.053093  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:18.026214  1236 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:23:27.061709  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:27.061784  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:37.067173  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:37.067613  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:47.069550  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:47.069739  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:23:57.071493  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:23:57.071651  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:07.087713  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:07.087857  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:17.090004  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:17.090096  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:19.041702  1236 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:24:27.108063  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:27.108158  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:37.110203  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:37.110306  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:47.112727  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:47.113052  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:24:57.117319  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:24:57.117856  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:07.121142  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:07.121464  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:17.133107  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:17.133289  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:20.074683  1236 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:25:27.136927  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:27.137018  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:37.139205  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:37.139407  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:47.141439  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:47.141743  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:25:57.143180  1235 raft_server.cpp:524] Term: 4, last_index index: 3, committed_index: 3, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:25:57.143265  1261 raft_server.h:59] Peer refresh succeeded!
I20220408 20:26:06.981592  1106 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:26:07.147325  1235 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:26:07.147382  1235 raft_server.cpp:716] Set shutting_down = true
I20220408 20:26:07.147388  1235 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:26:07.147392  1235 raft_server.cpp:726] Replication state shutdown, store sequence: 0
I20220408 20:26:07.147399  1235 raft_server.cpp:730] node->shutdown
I20220408 20:26:07.147418  1235 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 4 state LEADER
I20220408 20:26:07.147586  1235 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:26:07.147640  1235 raft_server.cpp:734] node->join
I20220408 20:26:07.147671  1261 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:26:07.147692  1261 raft_server.h:243] This node is down
I20220408 20:26:07.147723  1235 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 4 state SHUTDOWN
I20220408 20:26:07.147831  1235 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:26:07.147838  1235 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:26:07.148090  1235 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:26:07.148169  1235 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:26:07.149214  1235 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:26:07.149226  1235 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:26:08.086269  1236 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:26:08.086655  1236 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:26:08.088547  1235 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:26:08.090353  1235 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:26:08.092339  1106 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:26:08.092542  1106 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:26:08.092669  1106 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:26:08.092700  1106 typesense_server_utils.cpp:459] Deleting server
I20220408 20:26:08.092834  1106 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:26:08.093508  1106 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:26:08
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:26:08.147275  1570 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:26:08.147362  1570 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:26:08.147611  1570 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:26:08.152494  1570 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:26:08.191373  1570 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:26:08.204664  1570 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:26:08.204747  1708 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:26:08.204960  1570 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:26:08.206899  1709 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:26:08.213738  1708 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:26:08.213778  1708 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:26:08.214673  1709 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:26:08.215567  1708 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:26:08.215675  1708 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:26:08.216636  1708 store.h:283] rm /var/lib/typesense/db success
I20220408 20:26:08.216738  1708 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:26:08.231410  1708 store.h:307] DB open success!
I20220408 20:26:08.231427  1708 raft_server.cpp:469] Loading collections from disk...
I20220408 20:26:08.231444  1708 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:26:08.231487  1708 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:26:08.231492  1708 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:26:08.231495  1708 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:26:08.231958  1708 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:26:08.232532  1708 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:26:08.232558  1708 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:26:08.232874  1708 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:26:08.232935  1708 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 55
I20220408 20:26:08.232969  1708 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:26:08.233057  1708 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 4 votedfor 127.0.0.1:8107:8108 time: 17
I20220408 20:26:08.233115  1708 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 4 votedfor 127.0.0.1:8107:8108 time: 8
I20220408 20:26:08.233129  1708 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 4 last_log_id: (index=3,term=4) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:26:08.233171  1708 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 4 start vote and grant vote self
I20220408 20:26:08.234741  1708 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 5 votedfor 127.0.0.1:8107:8108 time: 1068
I20220408 20:26:08.234764  1708 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 5 become leader of group 127.0.0.1:8107:8108 
I20220408 20:26:08.234854  1708 raft_server.cpp:127] Node last_index: 4
I20220408 20:26:08.234863  1708 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:26:08.234879  1708 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:26:08.234887  1708 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:26:08.234899  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 0
W20220408 20:26:08.234905  1708 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:26:08.234927  1708 raft_server.cpp:578] Node not ready yet (known_applied_index is 0).
E20220408 20:26:08.234994  1717 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:26:08.235618  1716 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:26:08.235635  1716 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:26:08.235641  1716 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:26:08.235644  1716 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:26:08.235656  1716 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:26:08.235666  1716 raft_server.h:234] Node becomes leader, term: 5
I20220408 20:26:18.236305  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:26:18.236379  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:26:28.239228  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:26:28.239322  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:26:38.240679  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:26:38.240758  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:26:48.245383  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:26:48.245476  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:26:58.247135  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:26:58.247237  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:08.249107  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:08.249202  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:09.235839  1709 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:27:18.250710  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:18.250809  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:28.252562  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:28.252704  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:38.254770  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:38.254858  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:48.260690  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:48.262017  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:27:58.263594  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:27:58.263687  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:08.265084  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:08.265168  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:10.252784  1709 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:28:18.267870  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:18.268000  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:28.272123  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:28.272212  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:38.274518  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:38.274639  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:48.280123  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:48.280220  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:28:58.282897  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:28:58.283385  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:29:08.288502  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:29:08.288663  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:29:11.272447  1709 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:29:18.290838  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:29:18.291110  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:29:28.294396  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:29:28.294752  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:29:38.296610  1708 raft_server.cpp:524] Term: 5, last_index index: 4, committed_index: 4, known_applied_index: 4, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:29:38.296962  1716 raft_server.h:59] Peer refresh succeeded!
I20220408 20:29:45.589630  1570 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:29:46.298215  1708 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:29:46.298288  1708 raft_server.cpp:716] Set shutting_down = true
I20220408 20:29:46.298295  1708 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:29:46.298297  1708 raft_server.cpp:726] Replication state shutdown, store sequence: 0
I20220408 20:29:46.298312  1708 raft_server.cpp:730] node->shutdown
I20220408 20:29:46.298319  1708 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 5 state LEADER
I20220408 20:29:46.298530  1708 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:29:46.298561  1708 raft_server.cpp:734] node->join
I20220408 20:29:46.298610  1716 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:29:46.298630  1716 raft_server.h:243] This node is down
I20220408 20:29:46.298671  1708 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 5 state SHUTDOWN
I20220408 20:29:46.298759  1708 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:29:46.298764  1708 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:29:46.298970  1708 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:29:46.299022  1708 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:29:46.299926  1708 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:29:46.299937  1708 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:29:47.292739  1709 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:29:47.296041  1709 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:29:47.298353  1708 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:29:47.300022  1708 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:29:47.301548  1570 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:29:47.301630  1570 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:29:47.301708  1570 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:29:47.301740  1570 typesense_server_utils.cpp:459] Deleting server
I20220408 20:29:47.301844  1570 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:29:47.302392  1570 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:29:47
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:29:47.323313  1899 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:29:47.323385  1899 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:29:47.323632  1899 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:29:47.326388  1899 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:29:47.336475  1899 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:29:47.344684  1899 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:29:47.345007  1899 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:29:47.345129  2032 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:29:47.346719  2033 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:29:47.348791  2033 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:29:47.356299  2032 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:29:47.356353  2032 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:29:47.356645  2032 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:29:47.356773  2032 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:29:47.357434  2032 store.h:283] rm /var/lib/typesense/db success
I20220408 20:29:47.357508  2032 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:29:47.381417  2032 store.h:307] DB open success!
I20220408 20:29:47.381438  2032 raft_server.cpp:469] Loading collections from disk...
I20220408 20:29:47.381456  2032 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:29:47.381517  2032 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:29:47.381532  2032 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:29:47.381541  2032 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:29:47.382000  2032 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:29:47.382548  2032 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:29:47.382572  2032 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:29:47.382862  2032 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:29:47.382910  2032 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 37
I20220408 20:29:47.382941  2032 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:29:47.383030  2032 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 5 votedfor 127.0.0.1:8107:8108 time: 21
I20220408 20:29:47.383080  2032 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 5 votedfor 127.0.0.1:8107:8108 time: 4
I20220408 20:29:47.383086  2032 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 5 last_log_id: (index=4,term=5) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:29:47.383105  2032 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 5 start vote and grant vote self
I20220408 20:29:47.384647  2032 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 6 votedfor 127.0.0.1:8107:8108 time: 1280
I20220408 20:29:47.384672  2032 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 6 become leader of group 127.0.0.1:8107:8108 
I20220408 20:29:47.384712  2032 raft_server.cpp:127] Node last_index: 4
I20220408 20:29:47.384718  2032 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:29:47.384721  2032 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:29:47.384727  2032 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:29:47.384738  2032 raft_server.cpp:524] Term: 6, last_index index: 4, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 0
W20220408 20:29:47.384744  2032 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:29:47.384766  2032 raft_server.cpp:578] Node not ready yet (known_applied_index is 0).
E20220408 20:29:47.384868  2036 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:29:47.386020  2040 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:47.386045  2040 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:47.386054  2040 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:47.386060  2040 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:47.386065  2040 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:47.386106  2040 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:29:47.386121  2040 raft_server.h:234] Node becomes leader, term: 6
I20220408 20:29:51.262694  1899 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:29:51.385803  2032 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:29:51.385887  2032 raft_server.cpp:716] Set shutting_down = true
I20220408 20:29:51.385900  2032 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:29:51.385905  2032 raft_server.cpp:726] Replication state shutdown, store sequence: 0
I20220408 20:29:51.385926  2032 raft_server.cpp:730] node->shutdown
I20220408 20:29:51.385949  2032 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 6 state LEADER
I20220408 20:29:51.386196  2032 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:29:51.386247  2040 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:29:51.386260  2032 raft_server.cpp:734] node->join
I20220408 20:29:51.386315  2040 raft_server.h:243] This node is down
I20220408 20:29:51.386371  2032 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 6 state SHUTDOWN
I20220408 20:29:51.386540  2032 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:29:51.386548  2032 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:29:51.386763  2032 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:29:51.386889  2032 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:29:51.387821  2032 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:29:51.387836  2032 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:29:52.349570  2033 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:29:52.349756  2033 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:29:52.351857  2032 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:29:52.353683  2032 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:29:52.355090  1899 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:29:52.355365  1899 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:29:52.355439  1899 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:29:52.355460  1899 typesense_server_utils.cpp:459] Deleting server
I20220408 20:29:52.355571  1899 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:29:52.356137  1899 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:29:52
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:29:52.378582  2133 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:29:52.378660  2133 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:29:52.378983  2133 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:29:52.382212  2133 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:29:52.404244  2133 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:29:52.417373  2133 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:29:52.417722  2133 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:29:52.417866  2265 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:29:52.419374  2266 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:29:52.435724  2266 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:29:52.436653  2265 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:29:52.436674  2265 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:29:52.436849  2265 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:29:52.436978  2265 raft_server.cpp:104] Snapshot does not exist. We will remove db dir and init db fresh.
I20220408 20:29:52.437551  2265 store.h:283] rm /var/lib/typesense/db success
I20220408 20:29:52.437626  2265 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:29:52.445188  2265 store.h:307] DB open success!
I20220408 20:29:52.445206  2265 raft_server.cpp:469] Loading collections from disk...
I20220408 20:29:52.445230  2265 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:29:52.445276  2265 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:29:52.445283  2265 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:29:52.445288  2265 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:29:52.445806  2265 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:29:52.446208  2265 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:29:52.446233  2265 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:29:52.446591  2265 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:29:52.446652  2265 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 52
I20220408 20:29:52.446698  2265 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:29:52.446805  2265 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 6 votedfor 127.0.0.1:8107:8108 time: 15
I20220408 20:29:52.446868  2265 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 6 votedfor 127.0.0.1:8107:8108 time: 6
I20220408 20:29:52.446875  2265 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 6 last_log_id: (index=5,term=6) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:29:52.446907  2265 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 6 start vote and grant vote self
I20220408 20:29:52.448117  2265 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 7 votedfor 127.0.0.1:8107:8108 time: 827
I20220408 20:29:52.448135  2265 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 7 become leader of group 127.0.0.1:8107:8108 
I20220408 20:29:52.448202  2265 raft_server.cpp:127] Node last_index: 5
I20220408 20:29:52.448210  2265 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:29:52.448215  2265 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:29:52.448220  2265 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:29:52.448230  2265 raft_server.cpp:524] Term: 7, last_index index: 5, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 0
W20220408 20:29:52.448241  2265 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:29:52.448252  2265 raft_server.cpp:578] Node not ready yet (known_applied_index is 0).
E20220408 20:29:52.448324  2271 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:29:52.449352  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449378  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449386  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449393  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449398  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449404  2275 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:29:52.449416  2275 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:29:52.449429  2275 raft_server.h:234] Node becomes leader, term: 7
I20220408 20:30:02.450129  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:02.450304  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:12.455780  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:12.456066  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:22.458638  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:22.459281  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:32.471833  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:32.472137  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:42.475788  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:42.476009  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:52.478374  2265 raft_server.cpp:524] Term: 7, last_index index: 6, committed_index: 6, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0
I20220408 20:30:52.478684  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:30:53.459506  2266 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:30:53.927168  2275 node.cpp:911] node default_group:127.0.0.1:8107:8108 starts to do snapshot
I20220408 20:30:53.927935  2275 raft_server.cpp:424] on_snapshot_save
I20220408 20:30:53.928220  2275 batched_indexer.cpp:350] Serialized 0 in-flight requests for snapshot.
I20220408 20:30:53.938628  2275 raft_server.cpp:368] save_snapshot called
I20220408 20:30:53.939543  2275 snapshot.cpp:638] Deleting /var/lib/typesense/state/snapshot/snapshot_00000000000000000006
I20220408 20:30:53.939566  2275 snapshot.cpp:644] Renaming /var/lib/typesense/state/snapshot/temp to /var/lib/typesense/state/snapshot/snapshot_00000000000000000006
I20220408 20:30:53.939591  2275 snapshot_executor.cpp:234] node default_group:127.0.0.1:8107:8108 snapshot_save_done, last_included_index=6 last_included_term=7
I20220408 20:30:53.942710  2271 raft_server.cpp:692] Dummy write to http://127.0.0.1:8108/health, status = 200, response = {"ok":true}
I20220408 20:30:53.942734  2271 raft_server.cpp:417] save_snapshot done
I20220408 20:31:02.483748  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:02.483980  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:12.485329  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:12.485476  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:22.487634  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:22.488094  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:32.490674  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:32.492766  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:42.507158  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:42.507326  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:52.510133  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:31:52.510272  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:31:54.472190  2266 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:32:02.516582  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:02.516724  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:12.519152  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:12.519410  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:22.521055  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:22.521275  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:32.522949  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:32.523105  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:42.526093  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:42.526245  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:52.528911  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:32:52.529064  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:32:55.488020  2266 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:33:02.530371  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:02.530530  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:12.533510  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:12.533671  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:22.536895  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:22.537277  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:32.539045  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:32.539285  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:42.547051  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:42.547250  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:52.550364  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:33:52.550621  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:33:56.520933  2266 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:34:02.552517  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:02.552690  2271 raft_server.h:59] Peer refresh succeeded!
I20220408 20:34:12.553988  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:12.554163  2270 raft_server.h:59] Peer refresh succeeded!
I20220408 20:34:22.560783  2265 raft_server.cpp:524] Term: 7, last_index index: 7, committed_index: 7, known_applied_index: 7, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:22.560942  2275 raft_server.h:59] Peer refresh succeeded!
I20220408 20:34:22.650262  2133 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:34:23.561024  2265 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:34:23.561077  2265 raft_server.cpp:716] Set shutting_down = true
I20220408 20:34:23.561081  2265 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:34:23.561084  2265 raft_server.cpp:726] Replication state shutdown, store sequence: 3
I20220408 20:34:23.561090  2265 raft_server.cpp:730] node->shutdown
I20220408 20:34:23.561106  2265 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 7 state LEADER
I20220408 20:34:23.561300  2265 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:34:23.561331  2275 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:34:23.561345  2265 raft_server.cpp:734] node->join
I20220408 20:34:23.561372  2275 raft_server.h:243] This node is down
I20220408 20:34:23.561424  2265 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 7 state SHUTDOWN
I20220408 20:34:23.561754  2265 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:34:23.561767  2265 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:34:23.562063  2265 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:34:23.562196  2265 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:34:23.563549  2265 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:34:23.563573  2265 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:34:24.532536  2266 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:34:24.532816  2266 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:34:24.535809  2265 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:34:24.541664  2265 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:34:24.543913  2133 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:34:24.544140  2133 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:34:24.544229  2133 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:34:24.544256  2133 typesense_server_utils.cpp:459] Deleting server
I20220408 20:34:24.544385  2133 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:34:24.551528  2133 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:34:24
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:34:24.585935  2616 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:34:24.586033  2616 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:34:24.586295  2616 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:34:24.592226  2616 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:34:24.612305  2616 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:34:24.635020  2616 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:34:24.635110  2754 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:34:24.635509  2616 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:34:24.635941  2755 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:34:24.639562  2755 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:34:24.643496  2754 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:34:24.643537  2754 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:34:24.643757  2754 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:34:24.644184  2754 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:34:24.644261  2754 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 62
I20220408 20:34:24.644304  2754 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:34:24.644451  2754 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 7 votedfor 127.0.0.1:8107:8108 time: 25
I20220408 20:34:24.644861  2781 raft_server.cpp:490] on_snapshot_load
I20220408 20:34:24.645496  2781 store.h:283] rm /var/lib/typesense/db success
I20220408 20:34:24.645635  2781 store.h:293] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000006/db_snapshot to /var/lib/typesense/db success
I20220408 20:34:24.645645  2781 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:34:24.657147  2781 store.h:307] DB open success!
I20220408 20:34:24.657176  2781 raft_server.cpp:469] Loading collections from disk...
I20220408 20:34:24.657215  2781 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:34:24.657299  2781 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:34:24.657307  2781 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:34:24.657313  2781 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:34:24.658052  2781 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:34:24.658694  2781 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:34:24.658861  2781 batched_indexer.cpp:400] Restored 0 in-flight requests from snapshot.
I20220408 20:34:24.658871  2781 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:34:24.658900  2781 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:34:24.659056  2781 snapshot_executor.cpp:264] node default_group:127.0.0.1:8107:8108 snapshot_load_done, last_included_index: 6 last_included_term: 7 peers: "127.0.0.1:8107:8108"
I20220408 20:34:24.659351  2754 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 7 votedfor 127.0.0.1:8107:8108 time: 28
I20220408 20:34:24.659374  2754 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 7 last_log_id: (index=7,term=7) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:34:24.659412  2754 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 7 start vote and grant vote self
I20220408 20:34:24.664449  2754 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 8 votedfor 127.0.0.1:8107:8108 time: 4932
I20220408 20:34:24.664505  2754 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 8 become leader of group 127.0.0.1:8107:8108 
I20220408 20:34:24.664577  2754 raft_server.cpp:127] Node last_index: 7
I20220408 20:34:24.664585  2754 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:34:24.664590  2754 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:34:24.664597  2754 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:34:24.664616  2754 raft_server.cpp:524] Term: 8, last_index index: 7, committed_index: 0, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 1
W20220408 20:34:24.664625  2754 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:34:24.664744  2785 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:34:24.666589  2776 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:34:24.666745  2776 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:34:24.666765  2776 raft_server.h:234] Node becomes leader, term: 8
I20220408 20:34:28.640431  2616 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:34:28.665329  2754 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:34:28.665380  2754 raft_server.cpp:716] Set shutting_down = true
I20220408 20:34:28.665386  2754 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:34:28.665390  2754 raft_server.cpp:726] Replication state shutdown, store sequence: 3
I20220408 20:34:28.665396  2754 raft_server.cpp:730] node->shutdown
I20220408 20:34:28.665400  2754 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 8 state LEADER
I20220408 20:34:28.665446  2754 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:34:28.665457  2754 raft_server.cpp:734] node->join
I20220408 20:34:28.665524  2781 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:34:28.665582  2781 raft_server.h:243] This node is down
I20220408 20:34:28.665648  2754 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 8 state SHUTDOWN
I20220408 20:34:28.666024  2754 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:34:28.666039  2754 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:34:28.666280  2754 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:34:28.666836  2754 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:34:28.667201  2754 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:34:28.667209  2754 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:34:29.640413  2755 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:34:29.640610  2755 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:34:29.642084  2754 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:34:29.643461  2754 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:34:29.644712  2616 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:34:29.644881  2616 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:34:29.644898  2616 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:34:29.644903  2616 typesense_server_utils.cpp:459] Deleting server
I20220408 20:34:29.644945  2616 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:34:29.647516  2616 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:34:29
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:34:29.659396  2850 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:34:29.659480  2850 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:34:29.659703  2850 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:34:29.663554  2850 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:34:29.672652  2850 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:34:29.679761  2850 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:34:29.679849  2982 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:34:29.680552  2850 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:34:29.680554  2983 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:34:29.683279  2982 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:34:29.683310  2982 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:34:29.683465  2982 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:34:29.683487  2983 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:34:29.683751  2982 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:34:29.683791  2982 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 28
I20220408 20:34:29.683810  2982 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:34:29.683926  2982 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 8 votedfor 127.0.0.1:8107:8108 time: 11
I20220408 20:34:29.684274  2986 raft_server.cpp:490] on_snapshot_load
I20220408 20:34:29.684841  2986 store.h:283] rm /var/lib/typesense/db success
I20220408 20:34:29.684952  2986 store.h:293] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000006/db_snapshot to /var/lib/typesense/db success
I20220408 20:34:29.684962  2986 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:34:29.692390  2986 store.h:307] DB open success!
I20220408 20:34:29.692409  2986 raft_server.cpp:469] Loading collections from disk...
I20220408 20:34:29.692421  2986 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:34:29.692454  2986 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:34:29.692459  2986 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:34:29.692463  2986 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:34:29.692863  2986 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:34:29.693307  2986 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:34:29.693362  2986 batched_indexer.cpp:400] Restored 0 in-flight requests from snapshot.
I20220408 20:34:29.693368  2986 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:34:29.693382  2986 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:34:29.693423  2986 snapshot_executor.cpp:264] node default_group:127.0.0.1:8107:8108 snapshot_load_done, last_included_index: 6 last_included_term: 7 peers: "127.0.0.1:8107:8108"
I20220408 20:34:29.693502  2982 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 8 votedfor 127.0.0.1:8107:8108 time: 19
I20220408 20:34:29.693511  2982 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 8 last_log_id: (index=8,term=8) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:34:29.693537  2982 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 8 start vote and grant vote self
I20220408 20:34:29.694728  2982 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 9 votedfor 127.0.0.1:8107:8108 time: 942
I20220408 20:34:29.694751  2982 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 9 become leader of group 127.0.0.1:8107:8108 
I20220408 20:34:29.694789  2982 raft_server.cpp:127] Node last_index: 8
I20220408 20:34:29.694797  2982 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:34:29.694801  2982 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:34:29.694808  2982 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:34:29.694821  2982 raft_server.cpp:524] Term: 9, last_index index: 8, committed_index: 0, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 1
W20220408 20:34:29.694828  2982 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:34:29.694895  2986 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:34:29.696709  2993 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:34:29.696771  2993 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:34:29.696780  2993 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:34:29.696794  2993 raft_server.h:234] Node becomes leader, term: 9
I20220408 20:34:39.696528  2982 raft_server.cpp:524] Term: 9, last_index index: 9, committed_index: 9, known_applied_index: 9, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:39.696614  2986 raft_server.h:59] Peer refresh succeeded!
I20220408 20:34:49.698974  2982 raft_server.cpp:524] Term: 9, last_index index: 9, committed_index: 9, known_applied_index: 9, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:49.699483  2993 raft_server.h:59] Peer refresh succeeded!
I20220408 20:34:59.701747  2982 raft_server.cpp:524] Term: 9, last_index index: 9, committed_index: 9, known_applied_index: 9, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:34:59.701874  2986 raft_server.h:59] Peer refresh succeeded!
I20220408 20:35:09.712996  2982 raft_server.cpp:524] Term: 9, last_index index: 9, committed_index: 9, known_applied_index: 9, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:35:09.713217  2993 raft_server.h:59] Peer refresh succeeded!
I20220408 20:35:19.716477  2982 raft_server.cpp:524] Term: 9, last_index index: 9, committed_index: 9, known_applied_index: 9, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:35:19.716598  2986 raft_server.h:59] Peer refresh succeeded!
I20220408 20:35:26.955605  2850 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:35:27.721019  2982 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:35:27.721074  2982 raft_server.cpp:716] Set shutting_down = true
I20220408 20:35:27.721081  2982 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:35:27.721084  2982 raft_server.cpp:726] Replication state shutdown, store sequence: 3
I20220408 20:35:27.721091  2982 raft_server.cpp:730] node->shutdown
I20220408 20:35:27.721105  2982 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 9 state LEADER
I20220408 20:35:27.721225  2982 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:35:27.721278  2982 raft_server.cpp:734] node->join
I20220408 20:35:27.721277  2993 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:35:27.721302  2993 raft_server.h:243] This node is down
I20220408 20:35:27.721338  2982 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 9 state SHUTDOWN
I20220408 20:35:27.721508  2982 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:35:27.721518  2982 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:35:27.721756  2982 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:35:27.721931  2982 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:35:27.722704  2982 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:35:27.722714  2982 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:35:28.696820  2983 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:35:28.697000  2983 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:35:28.699079  2982 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:35:28.700873  2982 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:35:28.702960  2850 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:35:28.703068  2850 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:35:28.703204  2850 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:35:28.703227  2850 typesense_server_utils.cpp:459] Deleting server
I20220408 20:35:28.703598  2850 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:35:28.715548  2850 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:35:28
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:35:28.745121  3130 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:35:28.745213  3130 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:35:28.745606  3130 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:35:28.750074  3130 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:35:28.799592  3130 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:35:28.831693  3130 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:35:28.831934  3251 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:35:28.832247  3130 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:35:28.839987  3252 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:35:28.846320  3252 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:35:28.851609  3251 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:35:28.851650  3251 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:35:28.851872  3251 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:35:28.852351  3251 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:35:28.852393  3251 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 32
I20220408 20:35:28.852418  3251 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:35:28.852517  3251 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 9 votedfor 127.0.0.1:8107:8108 time: 16
I20220408 20:35:28.852926  3262 raft_server.cpp:490] on_snapshot_load
I20220408 20:35:28.853559  3262 store.h:283] rm /var/lib/typesense/db success
I20220408 20:35:28.853718  3262 store.h:293] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000006/db_snapshot to /var/lib/typesense/db success
I20220408 20:35:28.853729  3262 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:35:28.863492  3262 store.h:307] DB open success!
I20220408 20:35:28.863512  3262 raft_server.cpp:469] Loading collections from disk...
I20220408 20:35:28.863525  3262 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:35:28.863572  3262 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:35:28.863577  3262 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:35:28.863582  3262 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:35:28.868360  3262 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:35:28.869279  3262 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:35:28.869372  3262 batched_indexer.cpp:400] Restored 0 in-flight requests from snapshot.
I20220408 20:35:28.869382  3262 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:35:28.869398  3262 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:28.869506  3262 snapshot_executor.cpp:264] node default_group:127.0.0.1:8107:8108 snapshot_load_done, last_included_index: 6 last_included_term: 7 peers: "127.0.0.1:8107:8108"
I20220408 20:35:28.869706  3251 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 9 votedfor 127.0.0.1:8107:8108 time: 38
I20220408 20:35:28.869746  3251 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 9 last_log_id: (index=9,term=9) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:35:28.869781  3251 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 9 start vote and grant vote self
I20220408 20:35:28.871210  3251 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 10 votedfor 127.0.0.1:8107:8108 time: 1289
I20220408 20:35:28.871263  3251 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 10 become leader of group 127.0.0.1:8107:8108 
I20220408 20:35:28.871309  3251 raft_server.cpp:127] Node last_index: 9
I20220408 20:35:28.871316  3251 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:35:28.871321  3251 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:35:28.871328  3251 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:35:28.871342  3251 raft_server.cpp:524] Term: 10, last_index index: 9, committed_index: 0, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 1
W20220408 20:35:28.871352  3251 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:35:28.871412  3254 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:35:28.872344  3262 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:28.872380  3262 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:28.872392  3262 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:28.872412  3262 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:35:28.872428  3262 raft_server.h:234] Node becomes leader, term: 10
I20220408 20:35:31.870342  3130 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:35:31.871976  3251 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:35:31.872016  3251 raft_server.cpp:716] Set shutting_down = true
I20220408 20:35:31.872023  3251 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:35:31.872026  3251 raft_server.cpp:726] Replication state shutdown, store sequence: 3
I20220408 20:35:31.872047  3251 raft_server.cpp:730] node->shutdown
I20220408 20:35:31.872059  3251 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 10 state LEADER
I20220408 20:35:31.872263  3251 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:35:31.872309  3251 raft_server.cpp:734] node->join
I20220408 20:35:31.872392  3254 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:35:31.872478  3254 raft_server.h:243] This node is down
I20220408 20:35:31.872570  3251 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 10 state SHUTDOWN
I20220408 20:35:31.872799  3251 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:35:31.872807  3251 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:35:31.873072  3251 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:35:31.873162  3251 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:35:31.874126  3251 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:35:31.874147  3251 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:35:32.848829  3252 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:35:32.849164  3252 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:35:32.850831  3251 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:35:32.852142  3251 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:35:32.853569  3130 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:35:32.853688  3130 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:35:32.853792  3130 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:35:32.853821  3130 typesense_server_utils.cpp:459] Deleting server
I20220408 20:35:32.853941  3130 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:35:32.869654  3130 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:35:32
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:35:32.912710  3348 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:35:32.912801  3348 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:35:32.913089  3348 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:35:32.922196  3348 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:35:33.024566  3348 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:35:33.062361  3348 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:35:33.062590  3348 http_server.cpp:111] SSL cert refresh interval: 28800s
I20220408 20:35:33.062638  3483 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:35:33.062752  3484 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:35:33.064666  3484 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:35:33.069929  3348 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:35:33.074684  3483 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:35:33.074735  3483 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:35:33.076023  3483 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:35:33.077672  3483 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:35:33.077750  3483 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 55
I20220408 20:35:33.077791  3483 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:35:33.077929  3483 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 10 votedfor 127.0.0.1:8107:8108 time: 15
I20220408 20:35:33.079649  3514 raft_server.cpp:490] on_snapshot_load
I20220408 20:35:33.085737  3514 store.h:283] rm /var/lib/typesense/db success
I20220408 20:35:33.086282  3514 store.h:293] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000006/db_snapshot to /var/lib/typesense/db success
I20220408 20:35:33.086303  3514 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:35:33.118547  3514 store.h:307] DB open success!
I20220408 20:35:33.118571  3514 raft_server.cpp:469] Loading collections from disk...
I20220408 20:35:33.118594  3514 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:35:33.118660  3514 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:35:33.118666  3514 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:35:33.118672  3514 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:35:33.122206  3514 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:35:33.123067  3514 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:35:33.123195  3514 batched_indexer.cpp:400] Restored 0 in-flight requests from snapshot.
I20220408 20:35:33.123209  3514 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:35:33.123225  3514 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:33.123328  3514 snapshot_executor.cpp:264] node default_group:127.0.0.1:8107:8108 snapshot_load_done, last_included_index: 6 last_included_term: 7 peers: "127.0.0.1:8107:8108"
I20220408 20:35:33.123466  3483 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 10 votedfor 127.0.0.1:8107:8108 time: 27
I20220408 20:35:33.123487  3483 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 10 last_log_id: (index=10,term=10) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:35:33.123531  3483 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 10 start vote and grant vote self
I20220408 20:35:33.142436  3483 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 11 votedfor 127.0.0.1:8107:8108 time: 18744
I20220408 20:35:33.142506  3483 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 11 become leader of group 127.0.0.1:8107:8108 
I20220408 20:35:33.142575  3483 raft_server.cpp:127] Node last_index: 10
I20220408 20:35:33.142588  3483 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:35:33.142592  3483 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:35:33.142598  3483 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:35:33.142613  3483 raft_server.cpp:524] Term: 11, last_index index: 10, committed_index: 0, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 1
W20220408 20:35:33.142625  3483 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:35:33.143044  3511 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:35:33.149224  3514 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:33.149251  3514 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:33.149258  3514 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:33.149263  3514 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:35:33.149277  3514 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:35:33.149291  3514 raft_server.h:234] Node becomes leader, term: 11
I20220408 20:35:43.144186  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:35:43.144292  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:35:53.145716  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:35:53.145998  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:03.152326  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:03.152576  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:13.156656  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:13.156898  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:23.165700  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:23.166301  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:33.167979  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:33.168217  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:34.084734  3484 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:36:43.169677  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:43.169936  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:36:53.172055  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:36:53.172286  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:03.177906  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:03.178061  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:13.179495  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:13.179692  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:23.181218  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:23.181416  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:33.183255  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:33.183498  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:35.105626  3484 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:37:43.185381  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:43.185971  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:37:53.187609  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:37:53.187880  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:03.189247  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:03.189613  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:13.190858  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:13.191011  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:23.193347  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:23.193542  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:33.196686  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:33.199718  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:36.156951  3484 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:38:43.198724  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:43.198849  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:38:53.200702  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:38:53.200930  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:03.203135  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:03.203362  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:13.207818  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:13.208158  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:23.210206  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:23.210372  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:33.212076  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:33.212316  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:37.167948  3484 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:39:43.217865  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:43.218168  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:39:53.221899  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:39:53.222133  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:40:03.224154  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:40:03.224500  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:40:13.226012  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:40:13.226310  3511 raft_server.h:59] Peer refresh succeeded!
I20220408 20:40:23.230521  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:40:23.230726  3507 raft_server.h:59] Peer refresh succeeded!
I20220408 20:40:33.231995  3483 raft_server.cpp:524] Term: 11, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:40:33.232187  3514 raft_server.h:59] Peer refresh succeeded!
I20220408 20:40:38.191557  3484 batched_indexer.cpp:242] Running GC for aborted requests, req map size: 0
I20220408 20:40:39.434232  3348 typesense_server_utils.cpp:46] Stopping Typesense server...
I20220408 20:40:40.233420  3483 typesense_server_utils.cpp:292] Typesense peering service is going to quit.
I20220408 20:40:40.233470  3483 raft_server.cpp:716] Set shutting_down = true
I20220408 20:40:40.233474  3483 raft_server.cpp:720] Waiting for in-flight writes to finish...
I20220408 20:40:40.233477  3483 raft_server.cpp:726] Replication state shutdown, store sequence: 3
I20220408 20:40:40.233484  3483 raft_server.cpp:730] node->shutdown
I20220408 20:40:40.233536  3483 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 11 state LEADER
I20220408 20:40:40.233728  3483 replicator.cpp:1475] Group default_group Fail to find the next candidate
I20220408 20:40:40.233805  3483 raft_server.cpp:734] node->join
I20220408 20:40:40.233836  3511 raft_server.h:239] Node stepped down : Raft node is going to quit.
I20220408 20:40:40.233975  3511 raft_server.h:243] This node is down
I20220408 20:40:40.234055  3483 node.cpp:929] node default_group:127.0.0.1:8107:8108 shutdown, current_term 11 state SHUTDOWN
I20220408 20:40:40.234647  3483 typesense_server_utils.cpp:297] raft_server.stop()
I20220408 20:40:40.234660  3483 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20220408 20:40:40.234918  3483 typesense_server_utils.cpp:300] raft_server.join()
I20220408 20:40:40.235090  3483 typesense_server_utils.cpp:303] Typesense peering service has quit.
I20220408 20:40:40.236349  3483 typesense_server_utils.cpp:423] Shutting down batch indexer...
I20220408 20:40:40.236368  3483 typesense_server_utils.cpp:426] Waiting for batch indexing thread to be done...
I20220408 20:40:41.192111  3484 batched_indexer.cpp:274] Notifying batch indexer threads about shutdown...
I20220408 20:40:41.192476  3484 batched_indexer.cpp:282] Batched indexer threadpool shutdown...
I20220408 20:40:41.194705  3483 typesense_server_utils.cpp:429] Shutting down server_thread_pool
I20220408 20:40:41.200096  3483 typesense_server_utils.cpp:433] Shutting down app_thread_pool.
I20220408 20:40:41.202286  3348 typesense_server_utils.cpp:447] Typesense API service has quit.
I20220408 20:40:41.202332  3348 typesense_server_utils.cpp:451] Deleting batch indexer
I20220408 20:40:41.202386  3348 typesense_server_utils.cpp:455] CURL clean up
I20220408 20:40:41.202415  3348 typesense_server_utils.cpp:459] Deleting server
I20220408 20:40:41.202561  3348 typesense_server_utils.cpp:463] CollectionManager dispose, this might take some time...
I20220408 20:40:41.216320  3348 typesense_server_utils.cpp:467] Bye.
Log file created at: 2022/04/08 20:40:41
Running on machine: censored
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220408 20:40:41.271126  3701 typesense_server_utils.cpp:309] Starting Typesense 0.23.0.rc20
I20220408 20:40:41.271241  3701 typesense_server_utils.cpp:312] Typesense is using jemalloc.
I20220408 20:40:41.271554  3701 typesense_server_utils.cpp:361] Thread pool size: 32
I20220408 20:40:41.282445  3701 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:40:41.302887  3701 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/meta
I20220408 20:40:41.326385  3701 typesense_server_utils.cpp:440] Starting API service...
I20220408 20:40:41.326506  3843 typesense_server_utils.cpp:212] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20220408 20:40:41.326617  3701 http_server.cpp:111] SSL cert refresh interval: 28800s
I20220408 20:40:41.327579  3701 http_server.cpp:172] Typesense has started listening on port 8108
I20220408 20:40:41.328063  3844 batched_indexer.cpp:120] Starting batch indexer with 32 threads.
I20220408 20:40:41.331935  3844 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20220408 20:40:41.336419  3843 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20220408 20:40:41.336452  3843 server.cpp:1048] Check out http://censored:8107 in web browser.
I20220408 20:40:41.336614  3843 raft_server.cpp:65] Nodes configuration: 127.0.0.1:8107:8108
I20220408 20:40:41.337044  3843 log.cpp:674] Use crc32c as the checksum type of appending entries
I20220408 20:40:41.337087  3843 log.cpp:1158] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 1 time: 35
I20220408 20:40:41.337117  3843 log.cpp:1098] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20220408 20:40:41.337261  3843 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 11 votedfor 127.0.0.1:8107:8108 time: 16
I20220408 20:40:41.337679  3863 raft_server.cpp:490] on_snapshot_load
I20220408 20:40:41.338376  3863 store.h:283] rm /var/lib/typesense/db success
I20220408 20:40:41.338570  3863 store.h:293] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000006/db_snapshot to /var/lib/typesense/db success
I20220408 20:40:41.338585  3863 store.h:61] Initializing DB by opening state dir: /var/lib/typesense/db
I20220408 20:40:41.347087  3863 store.h:307] DB open success!
I20220408 20:40:41.347111  3863 raft_server.cpp:469] Loading collections from disk...
I20220408 20:40:41.347133  3863 collection_manager.cpp:127] CollectionManager::load()
I20220408 20:40:41.347199  3863 auth_manager.cpp:32] Indexing 0 API key(s) found on disk.
I20220408 20:40:41.347213  3863 collection_manager.cpp:147] Loading upto 16 collections in parallel, 1000 documents at a time.
I20220408 20:40:41.347220  3863 collection_manager.cpp:154] Found 0 collection(s) on disk.
I20220408 20:40:41.347759  3863 collection_manager.cpp:216] Loaded 0 collection(s).
I20220408 20:40:41.348412  3863 collection_manager.cpp:220] Initializing batched indexer from snapshot state...
I20220408 20:40:41.348493  3863 batched_indexer.cpp:400] Restored 0 in-flight requests from snapshot.
I20220408 20:40:41.348500  3863 raft_server.cpp:476] Finished loading collections from disk.
I20220408 20:40:41.348513  3863 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.348615  3863 snapshot_executor.cpp:264] node default_group:127.0.0.1:8107:8108 snapshot_load_done, last_included_index: 6 last_included_term: 7 peers: "127.0.0.1:8107:8108"
I20220408 20:40:41.348814  3843 raft_meta.cpp:521] Loaded single stable meta, path /var/lib/typesense/state/meta term 11 votedfor 127.0.0.1:8107:8108 time: 27
I20220408 20:40:41.348834  3843 node.cpp:576] node default_group:127.0.0.1:8107:8108 init, term: 11 last_log_id: (index=11,term=11) conf: 127.0.0.1:8107:8108 old_conf: 
I20220408 20:40:41.348865  3843 node.cpp:1549] node default_group:127.0.0.1:8107:8108 term 11 start vote and grant vote self
I20220408 20:40:41.351676  3843 raft_meta.cpp:546] Saved single stable meta, path /var/lib/typesense/state/meta term 12 votedfor 127.0.0.1:8107:8108 time: 2720
I20220408 20:40:41.351722  3843 node.cpp:1783] node default_group:127.0.0.1:8107:8108 term 12 become leader of group 127.0.0.1:8107:8108 
I20220408 20:40:41.351785  3843 raft_server.cpp:127] Node last_index: 11
I20220408 20:40:41.351794  3843 typesense_server_utils.cpp:263] Typesense peering service is running on 127.0.0.1:8107
I20220408 20:40:41.351799  3843 typesense_server_utils.cpp:264] Snapshot interval configured as: 3600s
W20220408 20:40:41.351804  3843 controller.cpp:1454] SIGINT was installed with 1
I20220408 20:40:41.351820  3843 raft_server.cpp:524] Term: 12, last_index index: 11, committed_index: 0, known_applied_index: 6, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 1
W20220408 20:40:41.351830  3843 node.cpp:811] [default_group:127.0.0.1:8107:8108 ] Refusing concurrent configuration changing
E20220408 20:40:41.351882  3863 raft_server.h:61] Peer refresh failed, error: Doing another configuration change
I20220408 20:40:41.353286  3875 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.353313  3875 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.353322  3875 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.353327  3875 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.353332  3875 raft_server.h:251] Configuration of this group is 127.0.0.1:8107:8108
I20220408 20:40:41.353348  3875 node.cpp:3142] node default_group:127.0.0.1:8107:8108 reset ConfigurationCtx, new_peers: 127.0.0.1:8107:8108, old_peers: 127.0.0.1:8107:8108
I20220408 20:40:41.353377  3875 raft_server.h:234] Node becomes leader, term: 12
I20220408 20:40:51.363317  3843 raft_server.cpp:524] Term: 12, last_index index: 12, committed_index: 12, known_applied_index: 12, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:40:51.363587  3863 raft_server.h:59] Peer refresh succeeded!
I20220408 20:41:01.365166  3843 raft_server.cpp:524] Term: 12, last_index index: 12, committed_index: 12, known_applied_index: 12, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:41:01.365329  3875 raft_server.h:59] Peer refresh succeeded!
I20220408 20:41:11.367316  3843 raft_server.cpp:524] Term: 12, last_index index: 12, committed_index: 12, known_applied_index: 12, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:41:11.367525  3863 raft_server.h:59] Peer refresh succeeded!
I20220408 20:41:21.373665  3843 raft_server.cpp:524] Term: 12, last_index index: 12, committed_index: 12, known_applied_index: 12, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:41:21.373906  3875 raft_server.h:59] Peer refresh succeeded!
I20220408 20:41:31.376588  3843 raft_server.cpp:524] Term: 12, last_index index: 12, committed_index: 12, known_applied_index: 12, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3
I20220408 20:41:31.376852  3863 raft_server.h:59] Peer refresh succeeded!

jasonbosco commented 2 years ago

@GoedendagMC This usually happens when RAM is exhausted and the OS is repeatedly killing the Typesense process to preserve itself. Could you check journalctl for any Out of Memory (OOM) messages?

GoedendagMC commented 2 years ago

No OOM messages. Plenty GBs of RAM left, this is a fresh install so no indexes yet.

kishorenc commented 2 years ago

@GoedendagMC

Is the actual issue that Typesense server keeps restarting? I see the following log:

I20220408 20:15:24.869729   937 typesense_server_utils.cpp:46] Stopping Typesense server...

This happens when the Typesense server is sent a SIGINT or SIGTERM signal. So it's catching that and shutting down.

May I know what OS and version of the OS you are running Typesense on?
Have you tried using the latest builds? E.g. 0.23.0.rc55

GoedendagMC commented 2 years ago

This is me manually restarting the service so you can see the errors persist. I am on Ubuntu 20.04 and the issue persists even with the latest build.

kishorenc commented 2 years ago

This message is not strictly an error, unless it keep occurring over and over again.

typesense-server[3701]: E20220408 20:40:41.351882 3863 raft_server.h:61] Peer refresh failed, error: Doing another configuration change

This can be logged when a server is starting up. On startup, Typesense creates in-memory indices of on-disk data. During this phase, the peer refresh could fail, but it will eventually succeed once the indices have been created. As you can see, after some time, it succeeds:

I20220408 20:18:21.870023   947 raft_server.h:59] Peer refresh succeeded!

GoedendagMC commented 2 years ago

Okay, thanks. But why can I not visit port 8107? Anything to do with the raft_server messages in the log? EDIT: I found this was because of the peering service being bound to 127.0.0.1, which only accepts internal connections. Is this how it should be?

GoedendagMC commented 2 years ago

Also, how can one upgrade the Typesense server (semi-)automatically when having installed it through a "DEB package on Ubuntu/Debian"?

kishorenc commented 2 years ago

@GoedendagMC

Port 8107 is bound only to internal IP since it's used for clustering so must be internal IP for security reasons.

To upgrade:

apt-get -o Dpkg::Options::="--force-confdef" \
        -o Dpkg::Options::="--force-confold" \
        -y install /path/to/typesense.deb

danthegoodman1 commented 1 year ago

I don't think it resolving after some time is acceptable

We get looping

W20230823 19:13:14.341240   166 node.cpp:811] [default_group:10.0.165.12:8107:8108 ] Refusing concurrent configuration changing
E20230823 19:13:14.341295   201 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
I20230823 19:13:24.342011   166 raft_server.cpp:546] Term: 10, last_index index: 866389, committed_index: 866389, known_applied_index: 845438, applying_in
dex: 864297, queued_writes: 11017, pending_queue_size: 0, local_sequence: 163089504
W20230823 19:13:24.342031   166 node.cpp:811] [default_group:10.0.165.12:8107:8108 ] Refusing concurrent configuration changing
E20230823 19:13:24.342067   201 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

at scale, so we'll need to find something other than typesense that can actually be rebooted quickly.

jasonbosco commented 1 year ago

You want to run Typesense in a multi-node HA setup, so that even if one node is restarted and is rebuilding the index, the other two nodes will continue servicing traffic, so there's zero downtime at the cluster level.

This model is common in in-memory datastores (eg: Redis, Typesense, etc), where the in-memory data structures need to be (re)built on process restart, by definition of it being an in-memory datastore. And building this in-memory index takes time depending on the size of your dataset.

Since you mentioned scale - we currently serve over 1.4 Billion searches per month on Typesense Cloud with this architecture, and we haven't found the startup time to be an issue in HA mode.

danthegoodman1 commented 1 year ago

Perhaps my response was a bit harsh as it was in the middle of typesense downtime. I don’t think we should be forced for it to decide that it doesn’t have peers, if it hasn’t any peers listed in the config.

We didn’t use clustering because we observed that when the pod restarts from being rescheduled and gets a new ip, the old nodes continue to use the old ip instead of just using the dns name provided in the config.

lots of Multi-node with no leader: refusing to reset peers. and can't do pre_vote as it is not in 10.0.165.152:8107:8108,10.0.166.192:8107:8108

But perhaps that another issue to raise. However I think it still stands we shouldnt be forced into raft, no reason to invoke the extra io if running one node. Maybe I’m missing something though?

danthegoodman1 commented 1 year ago

And to clarify, the problem we were having at scale is that one lagging node would impair the cluster, which caused more load. It didn’t seem like it would evict one based on bad performance. In clustered mode with 3vcpu it simply could not handle our write load, and more resources was not appropriate for how little write throughput that was

jasonbosco commented 1 year ago

We didn’t use clustering because we observed that when the pod restarts from being rescheduled and gets a new ip, the old nodes continue to use the old ip instead of just using the dns name provided in the config.

Here's the latest on this btw: https://github.com/typesense/typesense/issues/465#issuecomment-1689460898

However I think it still stands we shouldnt be forced into raft, no reason to invoke the extra io if running one node. Maybe I’m missing something though?

Startup times are unrelated to Raft. When a write is sent, it gets written to disk and then the in-memory index is updated. Then when the Typesense process is restarted, it reads the data from disk and builds the in-memory index again. The startup time you see is taken up by the in-memory index rebuilding process, not the read from disk.

Multi-node with no leader: refusing to reset peers. and can't do pre_vote as it is not in 10.0.165.152:8107:8108,10.0.166.192:8107:8108

You mentioned you're running a single node. But this seems to suggest that there are two nodes? Unless this is due to the stale IP issue. I'd recommend trying out the solution in the first paragraph above.

And to clarify, the problem we were having at scale is that one lagging node would impair the cluster, which caused more load. It didn’t seem like it would evict one based on bad performance. In clustered mode with 3vcpu it simply could not handle our write load, and more resources was not appropriate for how little write throughput that was

Here are a couple of ways to handle write lags: https://typesense.org/docs/guide/syncing-data-into-typesense.html#handling-http-503s

danthegoodman1 commented 1 year ago

Startup times are unrelated to Raft

Then why does the node spin doing Peer refresh failed, error: Doing another configuration change for 5 minutes when it has no peers? Why can't it just immedaitely start loading the indexes into memory?

You mentioned you're running a single node. But this seems to suggest that there are two nodes? Unless this is due to the stale IP issue

Those are from when we tried to do a 3 node cluster.

For the --reset-peers-on-error flag, assuming an error comes from when ever a peer is lost?

jasonbosco commented 1 year ago

Then why does the node spin doing Peer refresh failed, error: Doing another configuration change for 5 minutes when it has no peers? Why can't it just immedaitely start loading the indexes into memory?

In a non-HA setup, if the data is available in the data dir, the node does immediately start building the indices in memory when it starts up. If you look at RAM consumption it should steadily increase as the index is built.

The log message about peer refresh failed shows up because the node is considered ready only after the in-memory indices are fully built.

For the --reset-peers-on-error flag, assuming an error comes from when ever a peer is lost?

Didn't get that, could you rephrase?

danthegoodman1 commented 1 year ago

There’s not that much data, maybe a few hundred MB tops, should it normally take 5+ min for one core to build the index?

for the reset peers on error, is losing a peer considered an error that causes a reset?

jasonbosco commented 1 year ago

There’s not that much data, maybe a few hundred MB tops, should it normally take 5+ min for one core to build the index?

Yeah, that sounds about right. If you can put it on 4 cores, it should be faster.

for the reset peers on error, is losing a peer considered an error that causes a reset?

No, only if a peer reports an error status explicitly, then a reset happens.

danthegoodman1 commented 1 year ago

Alright thanks for the tip, maybe we just need to give it more cores

danthegoodman1 commented 1 year ago

@jasonbosco it doesn't seem like with clustering that losing one peer can reconstruct the cluster. After the new peer has come online, the leader is just:

W20230912 00:30:33.221688   202 replicator.cpp:297] Group default_group fail to issue RPC to 10.0.166.129:8107:8108 _consecutive_error_times=1281, [E2][10.0
.166.129:8107][E2]peer_id not exist [R1][E2][10.0.166.129:8107][E2]peer_id not exist [R2][E2][10.0.166.129:8107][E2]peer_id not exist [R3][E2][10.0.166.129:
8107][E2]peer_id not exist
I20230912 00:30:34.220350   166 raft_server.cpp:564] Term: 9, last_index index: 1400990, committed_index: 1400988, known_applied_index: 1330219, applying_in
dex: 1362743, queued_writes: 20815, pending_queue_size: 4, local_sequence: 254360200
W20230912 00:30:34.220367   166 node.cpp:843] [default_group:10.0.165.157:8107:8108 ] Refusing concurrent configuration changing
W20230912 00:30:35.731559   203 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.166.129:8107:8108 _consecutive_error_times=1291, [E2][10.0
.166.129:8107][E2]peer_id not exist [R1][E2][10.0.166.129:8107][E2]peer_id not exist [R2][E2][10.0.166.129:8107][E2]peer_id not exist [R3][E2][10.0.166.129:
8107][E2]peer_id not exist
E20230912 00:30:44.227648   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

It seems like the issue is that while the elected raft leader is waiting to the dead one to come online, it's consuming lots of memory very fast. Then it gets OOM killed, and this cycle is continuing...

danthegoodman1 commented 1 year ago

which is very weird because I'm getting a mix of all rate limited and circuit breaker open responses. No reads or writes are being accepted.

The IP that is failing to get the RPC command is the new pod that is taking ages to load the index.

It seems like the cluster is refusing to respond while the one node was down.

I also watched the raft cluster finally reconstruct, and the nodes start communicating, only to have the leader die from OOM which is incredibly frustrating.

danthegoodman1 commented 1 year ago

I can see other nodes following one (.30 node):

I20230912 00:55:12.364737   176 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 13 votedfor 10.0.165.30:8107:810
8 time: 2446
I20230912 00:55:12.367782   178 raft_server.h:288] Node starts following { leader_id=10.0.165.30:8107:8108, term=13, status=Follower receives message from n
ew leader with the same term.}
I20230912 00:55:14.114202   166 raft_server.cpp:564] Term: 13, last_index index: 1401991, committed_index: 1401990, known_applied_index: 1324847, applying_i
ndex: 1325290, queued_writes: 270, pending_queue_size: 0, local_sequence: 252906950
I20230912 00:55:24.125872   166 raft_server.cpp:564] Term: 13, last_index index: 1402011, committed_index: 1402010, known_applied_index: 1324847, applying_i
ndex: 1326534, queued_writes: 1008, pending_queue_size: 0, local_sequence: 252942956
I20230912 00:55:25.715075   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 859
I20230912 00:55:34.134660   166 raft_server.cpp:564] Term: 13, last_index index: 1402030, committed_index: 1402030, known_applied_index: 1324847, applying_i
ndex: 1327667, queued_writes: 1673, pending_queue_size: 0, local_sequence: 252979837
I20230912 00:55:44.145262   166 raft_server.cpp:564] Term: 13, last_index index: 1402042, committed_index: 1402042, known_applied_index: 1324847, applying_i
ndex: 1327931, queued_writes: 1800, pending_queue_size: 0, local_sequence: 252994939

However the leader is just spitting out that same error:

I20230912 00:56:01.529208   166 raft_server.cpp:564] Term: 13, last_index index: 1402052, committed_index: 1402052, known_applied_index: 1330219, applying_i
ndex: 1332487, queued_writes: 1204, pending_queue_size: 0, local_sequence: 253809000
E20230912 00:56:11.531541   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:56:21.534137   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:56:31.536897   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:56:41.538825   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:56:51.540774   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:57:01.543011   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:57:11.545421   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:57:21.548163   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:57:31.550351   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:57:41.552753   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
W20230912 00:56:01.529232   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:11.531489   166 raft_server.cpp:564] Term: 13, last_index index: 1402062, committed_index: 1402062, known_applied_index: 1330219, applying_i
ndex: 1332879, queued_writes: 1400, pending_queue_size: 0, local_sequence: 253828552
W20230912 00:56:11.531509   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:21.534086   166 raft_server.cpp:564] Term: 13, last_index index: 1402075, committed_index: 1402075, known_applied_index: 1330219, applying_i
ndex: 1333199, queued_writes: 1553, pending_queue_size: 0, local_sequence: 253842103
W20230912 00:56:21.534106   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:31.536832   166 raft_server.cpp:564] Term: 13, last_index index: 1402075, committed_index: 1402075, known_applied_index: 1330219, applying_i
ndex: 1333576, queued_writes: 1744, pending_queue_size: 0, local_sequence: 253861192
W20230912 00:56:31.536855   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:41.538774   166 raft_server.cpp:564] Term: 13, last_index index: 1402094, committed_index: 1402094, known_applied_index: 1330219, applying_i
ndex: 1334022, queued_writes: 1942, pending_queue_size: 0, local_sequence: 253888692
W20230912 00:56:41.538795   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:45.866214   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1500
I20230912 00:56:51.540722   166 raft_server.cpp:564] Term: 13, last_index index: 1402145, committed_index: 1402143, known_applied_index: 1330219, applying_i
ndex: 1334356, queued_writes: 2095, pending_queue_size: 6, local_sequence: 253908322
W20230912 00:56:51.540742   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:01.542963   166 raft_server.cpp:564] Term: 13, last_index index: 1402229, committed_index: 1402228, known_applied_index: 1330219, applying_i
ndex: 1334614, queued_writes: 2158, pending_queue_size: 6, local_sequence: 253932483
W20230912 00:57:01.542982   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:11.545356   166 raft_server.cpp:564] Term: 13, last_index index: 1402338, committed_index: 1402331, known_applied_index: 1330219, applying_i
ndex: 1334931, queued_writes: 2273, pending_queue_size: 7, local_sequence: 253956496
W20230912 00:57:11.545377   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:21.548105   166 raft_server.cpp:564] Term: 13, last_index index: 1402434, committed_index: 1402420, known_applied_index: 1330219, applying_i
ndex: 1335182, queued_writes: 2347, pending_queue_size: 15, local_sequence: 253977352
W20230912 00:57:21.548123   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:31.550284   166 raft_server.cpp:564] Term: 13, last_index index: 1402537, committed_index: 1402529, known_applied_index: 1330219, applying_i
ndex: 1335453, queued_writes: 2438, pending_queue_size: 9, local_sequence: 254002105
W20230912 00:57:31.550308   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:41.552685   166 raft_server.cpp:564] Term: 13, last_index index: 1402661, committed_index: 1402645, known_applied_index: 1330219, applying_i
ndex: 1335727, queued_writes: 2521, pending_queue_size: 18, local_sequence: 254026004
W20230912 00:57:41.552708   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:46.870649   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1897
E20230912 00:57:51.556546   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

At the same time, you can see that once the indexes load after restarting and they start trying to sort raft, the memory usage goes the the moon:

danthegoodman1 commented 1 year ago

It's just walls of this as they pin themselves to max memory:

ndex: 1333576, queued_writes: 1744, pending_queue_size: 0, local_sequence: 253861192
W20230912 00:56:31.536855   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:41.538774   166 raft_server.cpp:564] Term: 13, last_index index: 1402094, committed_index: 1402094, known_applied_index: 1330219, applying_i
ndex: 1334022, queued_writes: 1942, pending_queue_size: 0, local_sequence: 253888692
W20230912 00:56:41.538795   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:56:45.866214   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1500
I20230912 00:56:51.540722   166 raft_server.cpp:564] Term: 13, last_index index: 1402145, committed_index: 1402143, known_applied_index: 1330219, applying_i
ndex: 1334356, queued_writes: 2095, pending_queue_size: 6, local_sequence: 253908322
W20230912 00:56:51.540742   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:01.542963   166 raft_server.cpp:564] Term: 13, last_index index: 1402229, committed_index: 1402228, known_applied_index: 1330219, applying_i
ndex: 1334614, queued_writes: 2158, pending_queue_size: 6, local_sequence: 253932483
W20230912 00:57:01.542982   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:11.545356   166 raft_server.cpp:564] Term: 13, last_index index: 1402338, committed_index: 1402331, known_applied_index: 1330219, applying_i
ndex: 1334931, queued_writes: 2273, pending_queue_size: 7, local_sequence: 253956496
W20230912 00:57:11.545377   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:21.548105   166 raft_server.cpp:564] Term: 13, last_index index: 1402434, committed_index: 1402420, known_applied_index: 1330219, applying_i
ndex: 1335182, queued_writes: 2347, pending_queue_size: 15, local_sequence: 253977352
W20230912 00:57:21.548123   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:31.550284   166 raft_server.cpp:564] Term: 13, last_index index: 1402537, committed_index: 1402529, known_applied_index: 1330219, applying_i
ndex: 1335453, queued_writes: 2438, pending_queue_size: 9, local_sequence: 254002105
W20230912 00:57:31.550308   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:41.552685   166 raft_server.cpp:564] Term: 13, last_index index: 1402661, committed_index: 1402645, known_applied_index: 1330219, applying_i
ndex: 1335727, queued_writes: 2521, pending_queue_size: 18, local_sequence: 254026004
W20230912 00:57:41.552708   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:57:46.870649   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1897
E20230912 00:57:51.556546   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:01.558537   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:11.565976   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:21.568310   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:31.574784   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:41.577401   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:58:51.579993   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:59:01.582965   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:59:11.585721   208 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
I20230912 00:57:51.555069   166 raft_server.cpp:564] Term: 13, last_index index: 1402802, committed_index: 1402793, known_applied_index: 1330219, applying_i
ndex: 1336008, queued_writes: 2601, pending_queue_size: 9, local_sequence: 254048224
W20230912 00:57:51.556213   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:01.558480   166 raft_server.cpp:564] Term: 13, last_index index: 1402926, committed_index: 1402920, known_applied_index: 1330219, applying_i
ndex: 1336201, queued_writes: 2637, pending_queue_size: 7, local_sequence: 254070791
W20230912 00:58:01.558498   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:02.796701   207 log.cpp:536] close a full segment. Current first_index: 1393728 last_index: 1402940 raft_sync_segments: 0 will_sync: 1 path:
 /usr/share/typesense/data/state/log/log_00000000000001393728_00000000000001402940
I20230912 00:58:02.796851   207 log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000001393728' to `/usr/share/typesense/data
/state/log/log_00000000000001393728_00000000000001402940'
I20230912 00:58:02.796895   207 log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000001402941' with fd=2279
I20230912 00:58:11.565901   166 raft_server.cpp:564] Term: 13, last_index index: 1403060, committed_index: 1403057, known_applied_index: 1330219, applying_i
ndex: 1336467, queued_writes: 2719, pending_queue_size: 5, local_sequence: 254094216
W20230912 00:58:11.565927   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:21.568238   166 raft_server.cpp:564] Term: 13, last_index index: 1403187, committed_index: 1403181, known_applied_index: 1330219, applying_i
ndex: 1336738, queued_writes: 2808, pending_queue_size: 9, local_sequence: 254119466
W20230912 00:58:21.568259   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:31.574714   166 raft_server.cpp:564] Term: 13, last_index index: 1403329, committed_index: 1403327, known_applied_index: 1330219, applying_i
ndex: 1336911, queued_writes: 2837, pending_queue_size: 7, local_sequence: 254136663
W20230912 00:58:31.574740   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:41.577292   166 raft_server.cpp:564] Term: 13, last_index index: 1403468, committed_index: 1403461, known_applied_index: 1330219, applying_i
ndex: 1337234, queued_writes: 2967, pending_queue_size: 12, local_sequence: 254162516
W20230912 00:58:41.577329   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:58:47.877456   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 2256
I20230912 00:58:51.579933   166 raft_server.cpp:564] Term: 13, last_index index: 1403597, committed_index: 1403593, known_applied_index: 1330219, applying_i
ndex: 1337609, queued_writes: 3122, pending_queue_size: 13, local_sequence: 254187939
W20230912 00:58:51.579954   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:59:01.582906   166 raft_server.cpp:564] Term: 13, last_index index: 1403757, committed_index: 1403744, known_applied_index: 1330219, applying_i
ndex: 1337891, queued_writes: 3231, pending_queue_size: 18, local_sequence: 254210048
W20230912 00:59:01.582926   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
I20230912 00:59:11.585628   166 raft_server.cpp:564] Term: 13, last_index index: 1403934, committed_index: 1403926, known_applied_index: 1330219, applying_i
ndex: 1338186, queued_writes: 3358, pending_queue_size: 10, local_sequence: 254229022
W20230912 00:59:11.585661   166 node.cpp:843] [default_group:10.0.165.30:8107:8108 ] Refusing concurrent configuration changing
E20230912 00:59:21.588634   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:59:31.591300   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:59:41.593106   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 00:59:51.595306   209 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20230912 01:00:01.606367   206 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

danthegoodman1 commented 1 year ago

It seems like some nodes are responding now, but not all of them

danthegoodman1 commented 1 year ago

I tripled resources and while they got to raft faster, they still pin max memory usage while trying to sort raft out... this seems like a bug...

again the leader node is just constantly spitting out:

I20230912 01:23:35.481695   166 raft_server.cpp:564] Term: 16, last_index index: 1422489, committed_index: 1422489, known_applied_index: 1332121, applying_i
ndex: 1351533, queued_writes: 10608, pending_queue_size: 0, local_sequence: 254766261
W20230912 01:23:35.481716   166 node.cpp:843] [default_group:10.0.166.31:8107:8108 ] Refusing concurrent configuration changing
I20230912 01:23:45.485780   166 raft_server.cpp:564] Term: 16, last_index index: 1422494, committed_index: 1422493, known_applied_index: 1332121, applying_i
ndex: 1352516, queued_writes: 11195, pending_queue_size: 1, local_sequence: 254794482
W20230912 01:23:45.485795   166 node.cpp:843] [default_group:10.0.166.31:8107:8108 ] Refusing concurrent configuration changing
I20230912 01:23:53.292019   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 8683
I20230912 01:23:55.490999   166 raft_server.cpp:564] Term: 16, last_index index: 1422496, committed_index: 1422496, known_applied_index: 1332121, applying_i
ndex: 1353564, queued_writes: 11835, pending_queue_size: 0, local_sequence: 254821211
W20230912 01:23:55.491019   166 node.cpp:843] [default_group:10.0.166.31:8107:8108 ] Refusing concurrent configuration changing
I20230912 01:24:05.493762   166 raft_server.cpp:564] Term: 16, last_index index: 1422504, committed_index: 1422504, known_applied_index: 1332121, applying_i
ndex: 1355121, queued_writes: 12852, pending_queue_size: 0, local_sequence: 254848888
E20230912 01:24:15.499167   175 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

danthegoodman1 commented 1 year ago

And it only seems like the cluster starts working once this peer refresh failed stops... this only happens on the leader so I can't imagine it's about building the in memory indexes, because the instant it stopped all nodes started... and of course once they started working again they get OOM killed because they're using ungodly amounts of memory waiting for this peer refresh failed. It seems like no matter how many resources I throw at it, it decides to use all of it

danthegoodman1 commented 1 year ago

Is there some setting I can limit memory usage with? Watching kubernetes kill it because it decides to use 1 more byte than allowed of memory is dizzyingly frustrating

danthegoodman1 commented 1 year ago

I believe the high memory usage was due to a 100k healthy lag, I reduced to 10k and memory is falling during rebuild

danthegoodman1 commented 1 year ago

After getting it all sorted, I think the error logging just needs to be better so I know what actions I can take. Don't tell me what's wrong, tell me why it went wrong and maybe what I can do about it :)

typesense / typesense

Peer refresh failed, error: Doing another configuration change #573