search

p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Graceful materializer shutdown #626

Open adzialocha opened 2 months ago

adzialocha commented 2 months ago

Describe your changes here.

📋 Checklist

sandreae commented 2 months ago

I just tested out this branch and it appears that something about materialization seems broken now. Sessions complete successfully, entries are sent and received, but I don't see materialization tasks kick off, and when the next replication session occurs the same entries are sent.

Here are some logs: 

[2024-06-18T19:09:55Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f3cd78f31cd41fb554d605b11b8facdccba5d93322376ba988a4fee4d7893f43
[2024-06-18T19:09:55Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f8736e84afcf596185b75ce1e4f60255e427757fa09d982c843add786a1e5f5a
[2024-06-18T19:09:55Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020fd993d4aeac4610293cc27958ccc1257ac51994bed2893675acf532ecc5875d1
[2024-06-18T19:09:55Z TRACE aquadoggo::replication::manager] Message received: 0 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4) SyncDone(false)
[2024-06-18T19:09:55Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4)
[2024-06-18T19:09:59Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4)
[2024-06-18T19:09:59Z TRACE aquadoggo::replication::manager] Message received: 0 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4) Have([("297dce21fd3b4bd1e5ad3a557fd8ae617d3a72a75cada02abfdf7a96ee9fa17a", [(LogId(0), SeqNum(1)), (LogId(1), SeqNum(1)), (LogId(2), SeqNum(1)), (LogId(3), SeqNum(1)), (LogId(4), SeqNum(1)), (LogId(5), SeqNum(1)), (LogId(6), SeqNum(1)), (LogId(7), SeqNum(1)), (LogId(8), SeqNum(1)), (LogId(9), SeqNum(1)), (LogId(10), SeqNum(1)), (LogId(11), SeqNum(1)), (LogId(12), SeqNum(1)), (LogId(13), SeqNum(1)), (LogId(14), SeqNum(1)), (LogId(15), SeqNum(1)), (LogId(16), SeqNum(1)), (LogId(17), SeqNum(1)), (LogId(18), SeqNum(1)), (LogId(19), SeqNum(1)), (LogId(20), SeqNum(1)), (LogId(21), SeqNum(1)), (LogId(22), SeqNum(1)), (LogId(23), SeqNum(1)), (LogId(24), SeqNum(1)), (LogId(25), SeqNum(1)), (LogId(26), SeqNum(1)), (LogId(27), SeqNum(1)), (LogId(28), SeqNum(1)), (LogId(29), SeqNum(1)), (LogId(30), SeqNum(1)), (LogId(31), SeqNum(1)), (LogId(32), SeqNum(1)), (LogId(33), SeqNum(1)), (LogId(34), SeqNum(1)), (LogId(35), SeqNum(1)), (LogId(36), SeqNum(1)), (LogId(37), SeqNum(1)), (LogId(38), SeqNum(1)), (LogId(39), SeqNum(1)), (LogId(40), SeqNum(1)), (LogId(41), SeqNum(1)), (LogId(42), SeqNum(1)), (LogId(43), SeqNum(1)), (LogId(44), SeqNum(1)), (LogId(45), SeqNum(1)), (LogId(46), SeqNum(1)), (LogId(47), SeqNum(1)), (LogId(48), SeqNum(1)), (LogId(49), SeqNum(1))]), ("47b38aa6d2f51dbe6191a4e1ba2d5a88f7534272af68d009be00ea3d3f4bf9bf", [(LogId(0), SeqNum(1)), (LogId(1), SeqNum(1)), (LogId(2), SeqNum(2)), (LogId(3), SeqNum(1))])])
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00200461a351ae57e85ef2134eba71ac2fad658db4090da766c33f9f9cb084ec4cf8
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002006bfb3c30c994843c006b4311152daa2e7de668a439055b5f9899b203a64d560
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201b5aa5fe170560ca3d4b0e90f9cbd9d7f133ed3e7242eddd1387b6c512c38bbb
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201b6b7125036be36d832632bedf0de7552ed6566a7b742b6f26e902a0fba12435
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201fbcf6d22712ee0df88c3c5b4d4fb4bdbf474736261b5fe2a67aab44ca1a8c6d
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002021849ffe9546354fab1c421a6e5f5aaa49ae6d1bddffbe057444f67ba6e4743f
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020226bb1d6281244467fb33675f0b6ce871b51948de5181b719599349610651d39
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00202484d05c90f894183bfdca129b04bbebcbdda354e44c6f4ec48dc84439effb41
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00202a2f65b79de2f10a03fbd2adeeea491bfb36dd86998af86f8335414a654dbde0
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00203634b94bce6d03e7fa4b6aa8423c6bf8d010f4f29608cdc97048d6d1b504541e
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020425f0bc0520c9991006816e8d052c642f4bbade1ea31882230ccd9f9c4fda64f
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002043c46d8f5cf9dd2e4c6a7a160ce87887cf96ae62c219d66fe817793bea79040e
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002044eb3669c9011c1d662d7e06b1b28eac35fbd134c9df286860b3e797042b38bd
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00204947039d87f4d7af1bbc1245c99933b1b15d1563e5ec19d5e25a11e0018146e2
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207b8419615d96212f674de4ae191b13f02d4de6ac87cb85240f972c68986161cc
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020a960a5e60c7daf66f9b9056de6e7904247360017dd09c223800e65223c0adafe
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002054cc9ef33540e3ee7207b7258f5c1410dbb5c739fbb324cda553aca74062b5f4
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002055a48c1e76981a0ac5e1fa4efcdb47246b560c21b77b47baba61cdd9c7809360
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020572bac054cdd43f64b486ce9bfc96443160680f9ddd25e48717a4ba04684e345
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00205a889e0bcedc72e5aa240bbd6ffe87eda90185d5f8a9923c3be9a600ba62b9e7
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00205a94ce2313c2ab632545a2f432336e960a8fed1c307857b76491c3a9e771f8b8
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002064feb6c43eca60974c8f5e60475b1b931cdc920e8441bd4c88f280a0b2f0cedc
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207889f44a73c94bb9f7e7d087e47d87ab4854c3693bca35bf16c596c8c91c5fe7
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207f1b4d7115e518ed754024db27c86756479c9448aebc3f39a0e67c18b611bc25
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00208ad3be2b46ad202ec0a27bb6431152db17d603489ad8b5ae5dab85772a370344
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00208ced5b9fc23a3f3e87444c4be7f305bbfcf4989d77cf6d208c579de1fe0a3b79
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020975d97897bed56058b81d3ae87b7b662b69506af15d0ad986589306658e07e5a
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002098b9c13e1162b360a528196c4293ed1e00e71359048323e9af2929ddf1e30313
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020aae5589e5366943d1ead02c0863a5d5010cd68569ba8fdb69f10c43736f1940f
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020abf0e567ec55d407c288f7d77bd58f1a529fdb0eb6cbb74279b83b53fb4fa9c2
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020adf1b7947d7f21ea33ee53288558ce64bf8344805a11dc97dddc58fc235c21b0
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020aeaca910c8a3f5fb0f80f7e8e6878720272d3bdcefa45c97cfc627e3b5e4252c
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b07aff1450f742f479d34c4c05e4c13128357dc0b6b78764bac5dd5331a775b1
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b0c62328cf6c0418e0d0487e1d542a02a12ebbce750dff70f3667015c3b93b61
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b2281e448560941934599e14d78669ccb99488edc5d339c55b6071b132a2b05f
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b45d52e5129b359a742a7a4f28d5d9d1e2ead32f263055fde25cf8d0e333ac49
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b8cb971fa0157b8bfee1a2921a8fdfe35ba5ff23d99b2c47fda8b14e1cb06bf2
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020bdb78e578befd97784e2ec7710f5314fc22fb484e59eef7a66f948ea953a38dc
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020bf3850c2e693a46116039614aca3354b6b18f6546d3f37f3b625770df58c82cc
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020c67a6080082f0ec1e5e560b5966677bb09881cebe808cd80721bd2f0189b8ac8
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020c790691e036ed090392b4c06eb3586eb130edaee6ece2d4a48607f240bc11f91
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020ced7475f1083cede0f82ced26c70a9f3c9d693e09af1774723de6cf76b70a282
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020d4d9006aaac916e76793dd779bd3ee730820fd1ee9c6caa1d982e78bf6244fc3
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020d52544b10282eb8c4531338a77ba15b43dee69c35835bdeebb7738e9ba888145
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020dbd6a5b06075624a1638461b8b76bfc7c3efa6c768cee8ad016b156616c7795e
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020df662f01bd4eed879ebb2128edd3e0b55902f179eeaf8978e58011f96b488717
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020e1567cb6f7e097b449cd05174f96ac17f774d8b80ffea423c4d4b386e423cb0a
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020eb32d5af9f31efabcc947a26cb39a65872d482f67a00725c2d3ca1d968ad5b3e
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020ebf2746448b4fdf563de1486efc082c1243522d2579aeaaf12e7937c6bc86eba
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020edc73122071ac3f023cdf84c289f65cb339927fb14169833306d830e8c466339
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020eff38ee4ed5bed12b61452c6472f8cd9c692e4f39b36a6b35f16fde309d56d00
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f269f21f5d430480fbd386ad03dacd4632125177c2efba480aa7cf8f1ea9a6f3
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f3cd78f31cd41fb554d605b11b8facdccba5d93322376ba988a4fee4d7893f43
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f8736e84afcf596185b75ce1e4f60255e427757fa09d982c843add786a1e5f5a
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020fd993d4aeac4610293cc27958ccc1257ac51994bed2893675acf532ecc5875d1
[2024-06-18T19:10:00Z TRACE aquadoggo::replication::manager] Message received: 0 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4) SyncDone(false)
[2024-06-18T19:10:00Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4)
[2024-06-18T19:10:04Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4)
[2024-06-18T19:10:04Z TRACE aquadoggo::replication::manager] Message received: 0 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4) Have([("297dce21fd3b4bd1e5ad3a557fd8ae617d3a72a75cada02abfdf7a96ee9fa17a", [(LogId(0), SeqNum(1)), (LogId(1), SeqNum(1)), (LogId(2), SeqNum(1)), (LogId(3), SeqNum(1)), (LogId(4), SeqNum(1)), (LogId(5), SeqNum(1)), (LogId(6), SeqNum(1)), (LogId(7), SeqNum(1)), (LogId(8), SeqNum(1)), (LogId(9), SeqNum(1)), (LogId(10), SeqNum(1)), (LogId(11), SeqNum(1)), (LogId(12), SeqNum(1)), (LogId(13), SeqNum(1)), (LogId(14), SeqNum(1)), (LogId(15), SeqNum(1)), (LogId(16), SeqNum(1)), (LogId(17), SeqNum(1)), (LogId(18), SeqNum(1)), (LogId(19), SeqNum(1)), (LogId(20), SeqNum(1)), (LogId(21), SeqNum(1)), (LogId(22), SeqNum(1)), (LogId(23), SeqNum(1)), (LogId(24), SeqNum(1)), (LogId(25), SeqNum(1)), (LogId(26), SeqNum(1)), (LogId(27), SeqNum(1)), (LogId(28), SeqNum(1)), (LogId(29), SeqNum(1)), (LogId(30), SeqNum(1)), (LogId(31), SeqNum(1)), (LogId(32), SeqNum(1)), (LogId(33), SeqNum(1)), (LogId(34), SeqNum(1)), (LogId(35), SeqNum(1)), (LogId(36), SeqNum(1)), (LogId(37), SeqNum(1)), (LogId(38), SeqNum(1)), (LogId(39), SeqNum(1)), (LogId(40), SeqNum(1)), (LogId(41), SeqNum(1)), (LogId(42), SeqNum(1)), (LogId(43), SeqNum(1)), (LogId(44), SeqNum(1)), (LogId(45), SeqNum(1)), (LogId(46), SeqNum(1)), (LogId(47), SeqNum(1)), (LogId(48), SeqNum(1)), (LogId(49), SeqNum(1))]), ("47b38aa6d2f51dbe6191a4e1ba2d5a88f7534272af68d009be00ea3d3f4bf9bf", [(LogId(0), SeqNum(1)), (LogId(1), SeqNum(1)), (LogId(2), SeqNum(2)), (LogId(3), SeqNum(1))])])
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00200461a351ae57e85ef2134eba71ac2fad658db4090da766c33f9f9cb084ec4cf8
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002006bfb3c30c994843c006b4311152daa2e7de668a439055b5f9899b203a64d560
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201b5aa5fe170560ca3d4b0e90f9cbd9d7f133ed3e7242eddd1387b6c512c38bbb
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201b6b7125036be36d832632bedf0de7552ed6566a7b742b6f26e902a0fba12435
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00201fbcf6d22712ee0df88c3c5b4d4fb4bdbf474736261b5fe2a67aab44ca1a8c6d
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002021849ffe9546354fab1c421a6e5f5aaa49ae6d1bddffbe057444f67ba6e4743f
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020226bb1d6281244467fb33675f0b6ce871b51948de5181b719599349610651d39
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00202484d05c90f894183bfdca129b04bbebcbdda354e44c6f4ec48dc84439effb41
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00202a2f65b79de2f10a03fbd2adeeea491bfb36dd86998af86f8335414a654dbde0
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00203634b94bce6d03e7fa4b6aa8423c6bf8d010f4f29608cdc97048d6d1b504541e
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020425f0bc0520c9991006816e8d052c642f4bbade1ea31882230ccd9f9c4fda64f
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002043c46d8f5cf9dd2e4c6a7a160ce87887cf96ae62c219d66fe817793bea79040e
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002044eb3669c9011c1d662d7e06b1b28eac35fbd134c9df286860b3e797042b38bd
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00204947039d87f4d7af1bbc1245c99933b1b15d1563e5ec19d5e25a11e0018146e2
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207b8419615d96212f674de4ae191b13f02d4de6ac87cb85240f972c68986161cc
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020a960a5e60c7daf66f9b9056de6e7904247360017dd09c223800e65223c0adafe
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002054cc9ef33540e3ee7207b7258f5c1410dbb5c739fbb324cda553aca74062b5f4
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002055a48c1e76981a0ac5e1fa4efcdb47246b560c21b77b47baba61cdd9c7809360
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020572bac054cdd43f64b486ce9bfc96443160680f9ddd25e48717a4ba04684e345
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00205a889e0bcedc72e5aa240bbd6ffe87eda90185d5f8a9923c3be9a600ba62b9e7
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00205a94ce2313c2ab632545a2f432336e960a8fed1c307857b76491c3a9e771f8b8
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002064feb6c43eca60974c8f5e60475b1b931cdc920e8441bd4c88f280a0b2f0cedc
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207889f44a73c94bb9f7e7d087e47d87ab4854c3693bca35bf16c596c8c91c5fe7
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00207f1b4d7115e518ed754024db27c86756479c9448aebc3f39a0e67c18b611bc25
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00208ad3be2b46ad202ec0a27bb6431152db17d603489ad8b5ae5dab85772a370344
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 00208ced5b9fc23a3f3e87444c4be7f305bbfcf4989d77cf6d208c579de1fe0a3b79
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020975d97897bed56058b81d3ae87b7b662b69506af15d0ad986589306658e07e5a
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 002098b9c13e1162b360a528196c4293ed1e00e71359048323e9af2929ddf1e30313
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020aae5589e5366943d1ead02c0863a5d5010cd68569ba8fdb69f10c43736f1940f
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020abf0e567ec55d407c288f7d77bd58f1a529fdb0eb6cbb74279b83b53fb4fa9c2
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020adf1b7947d7f21ea33ee53288558ce64bf8344805a11dc97dddc58fc235c21b0
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020aeaca910c8a3f5fb0f80f7e8e6878720272d3bdcefa45c97cfc627e3b5e4252c
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b07aff1450f742f479d34c4c05e4c13128357dc0b6b78764bac5dd5331a775b1
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b0c62328cf6c0418e0d0487e1d542a02a12ebbce750dff70f3667015c3b93b61
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b2281e448560941934599e14d78669ccb99488edc5d339c55b6071b132a2b05f
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b45d52e5129b359a742a7a4f28d5d9d1e2ead32f263055fde25cf8d0e333ac49
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020b8cb971fa0157b8bfee1a2921a8fdfe35ba5ff23d99b2c47fda8b14e1cb06bf2
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020bdb78e578befd97784e2ec7710f5314fc22fb484e59eef7a66f948ea953a38dc
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020bf3850c2e693a46116039614aca3354b6b18f6546d3f37f3b625770df58c82cc
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020c67a6080082f0ec1e5e560b5966677bb09881cebe808cd80721bd2f0189b8ac8
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020c790691e036ed090392b4c06eb3586eb130edaee6ece2d4a48607f240bc11f91
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020ced7475f1083cede0f82ced26c70a9f3c9d693e09af1774723de6cf76b70a282
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020d4d9006aaac916e76793dd779bd3ee730820fd1ee9c6caa1d982e78bf6244fc3
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020d52544b10282eb8c4531338a77ba15b43dee69c35835bdeebb7738e9ba888145
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020dbd6a5b06075624a1638461b8b76bfc7c3efa6c768cee8ad016b156616c7795e
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020df662f01bd4eed879ebb2128edd3e0b55902f179eeaf8978e58011f96b488717
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020e1567cb6f7e097b449cd05174f96ac17f774d8b80ffea423c4d4b386e423cb0a
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020eb32d5af9f31efabcc947a26cb39a65872d482f67a00725c2d3ca1d968ad5b3e
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020ebf2746448b4fdf563de1486efc082c1243522d2579aeaaf12e7937c6bc86eba
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020edc73122071ac3f023cdf84c289f65cb339927fb14169833306d830e8c466339
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020eff38ee4ed5bed12b61452c6472f8cd9c692e4f39b36a6b35f16fde309d56d00
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f269f21f5d430480fbd386ad03dacd4632125177c2efba480aa7cf8f1ea9a6f3
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f3cd78f31cd41fb554d605b11b8facdccba5d93322376ba988a4fee4d7893f43
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020f8736e84afcf596185b75ce1e4f60255e427757fa09d982c843add786a1e5f5a
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::ingest] Received entry and operation: 0020fd993d4aeac4610293cc27958ccc1257ac51994bed2893675acf532ecc5875d1
[2024-06-18T19:10:05Z TRACE aquadoggo::replication::manager] Message received: 0 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4) SyncDone(false)
[2024-06-18T19:10:05Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWCKPiD97YoVZx7fYpLwT6UtCizxXbzBvTvbSB5fC46L2F (4)
^C[2024-06-18T19:10:06Z INFO  aquadoggo::manager] Received shutdown signal