search

canonical / dqlite

Embeddable, replicated and fault-tolerant SQL engine.
https://dqlite.io
Other
3.77k stars 213 forks source link

Nested barriers #544

Open MathieuBordere opened 7 months ago

MathieuBordere commented 7 months ago

It's possible for dqlite to request a raft_barrier through leader_barrier while a raft_barrier is already in progress.

It happens in one case where a new raft leader tries to commit a barrier entry to determine its commit index and at that moment dqlite tries to handle a query, notices its state machine is not up to date and fires a barrier.

I'm worried this could also lead to an avalanche of barriers on a busy system and raft will be more busy replicating barriers than actual transaction data.

log

LIBRAFT[285040] 2023-12-04T10:16:21.885837733 285058 DEBUG   convertToLeader      src/convert.c:201 become leader for term 2
LIBRAFT[285040] 2023-12-04T10:16:21.885842071 285058 DEBUG   convertClearCandidate src/convert.c:56  clear candidate state
LIBRAFT[285040] 2023-12-04T10:16:21.885846519 285058 DEBUG   convertSetState      src/convert.c:25  old_state:2 new_state:3
LIBRAFT[285040] 2023-12-04T10:16:21.885852010 285058 DEBUG   raft_barrier         src/client.c:90  barrier starting at 6
LIBRAFT[285040] 2023-12-04T10:16:21.885857360 285058 DEBUG   uvAppendEnqueueRequest src/uv_append.c:614 enqueue 1 entries
LIBRAFT[285040] 2023-12-04T10:16:21.885861708 285058 DEBUG   uvAppendEnqueueRequest src/uv_append.c:646 set uv->append_next_index 7
LIBRAFT[285040] 2023-12-04T10:16:21.885878710 285058 DEBUG   sendAppendEntries    src/replication.c:100 send 1 entries starting at 5 to server 1 (last index 6)
LIBRAFT[285040] 2023-12-04T10:16:21.885883719 285058 DEBUG   uvClientSend         src/uv_send.c:218 connection available -> write message
LIBRAFT[285040] 2023-12-04T10:16:21.885909899 285041 DEBUG   recvAppendEntries    src/recv_append_entries.c:37  self:1 from:2@@2 leader_commit:4 n_entries:1 pr
ev_log_index:5 prev_log_term:1, term:2
LIBRAFT[285040] 2023-12-04T10:16:21.885919487 285041 DEBUG   uvAppendEnqueueRequest src/uv_append.c:614 enqueue 1 entries
LIBRAFT[285040] 2023-12-04T10:16:21.885924797 285041 DEBUG   uvAppendEnqueueRequest src/uv_append.c:646 set uv->append_next_index 7
LIBDQLITE[285040] 2023-12-04T10:16:21.885950075 285041 DEBUG   gateway__resume      src/gateway.c:1448 gateway resume - finished
LIBDQLITE[285040] 2023-12-04T10:16:21.885957559 285040 DEBUG   clientClose          src/client/protocol.c:332 client close
LIBDQLITE[285040] 2023-12-04T10:16:21.885970894 285040 DEBUG   clientSendHandshake  src/client/protocol.c:352 client send handshake
LIBDQLITE[285040] 2023-12-04T10:16:21.885975102 285041 DEBUG   read_message_cb      src/conn.c:196 read error -4095
LIBDQLITE[285040] 2023-12-04T10:16:21.885980672 285041 DEBUG   conn__stop           src/conn.c:342 conn stop
LIBDQLITE[285040] 2023-12-04T10:16:21.885987044 285041 DEBUG   gateway__close       src/gateway.c:86  gateway close
LIBDQLITE[285040] 2023-12-04T10:16:21.885981263 285040 DEBUG   clientSendOpen       src/client/protocol.c:518 client send open name test
LIBDQLITE[285040] 2023-12-04T10:16:21.886005369 285040 DEBUG   clientRecvDb         src/client/protocol.c:530 client recvdb
LIBDQLITE[285040] 2023-12-04T10:16:21.886013304 285058 DEBUG   gateway__handle      src/gateway.c:1389 gateway handle
LIBDQLITE[285040] 2023-12-04T10:16:21.886020918 285058 DEBUG   handle_open          src/gateway.c:265 handle open
LIBDQLITE[285040] 2023-12-04T10:16:21.886026419 285058 DEBUG   leader__init         src/leader.c:126 leader init
LIBDQLITE[285040] 2023-12-04T10:16:21.886030977 285058 DEBUG   openConnection       src/leader.c:30  open connection filename test
LIBDQLITE[285040] 2023-12-04T10:16:21.886119364 285058 DEBUG   gateway__resume      src/gateway.c:1448 gateway resume - finished
LIBDQLITE[285040] 2023-12-04T10:16:21.886144802 285040 DEBUG   clientSendPrepare    src/client/protocol.c:543 client send prepare
LIBDQLITE[285040] 2023-12-04T10:16:21.886173366 285058 DEBUG   gateway__handle      src/gateway.c:1389 gateway handle
LIBDQLITE[285040] 2023-12-04T10:16:21.886179187 285058 DEBUG   handle_prepare       src/gateway.c:372 handle prepare
LIBDQLITE[285040] 2023-12-04T10:16:21.886192762 285058 DEBUG   leader__barrier      src/leader.c:466 leader barrier
LIBRAFT[285040] 2023-12-04T10:16:21.886198984 285058 DEBUG   raft_barrier         src/client.c:90  barrier starting at 7