search

dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.47k stars 920 forks source link

Dragonfly freezes if replica freezes #2191

Open chakaz opened 10 months ago

chakaz commented 10 months ago

I ran Dragonfly, attached a replica, and then pushed some data into the master. When doing so, I froze the replica (ctrl-c inside gdb, so the connection is still open). I noticed that the master Dragonfly is partially frozen, in that it does respond to PING, but a simple SET x y waits forever.

More details:

Here's the output of DEBUG STACKTRACE:

I20231119 14:32:38.991734 2564752 scheduler.cc:713] ------------ Fiber main (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da04564d  util::fb2::ProactorBase::Run()
I20231119 14:32:38.995779 2564752 scheduler.cc:713] ------------ Fiber _dispatch (ready) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da088554  util::fb2::UringProactor::MainLoop()
I20231119 14:32:38.996827 2564752 scheduler.cc:713] ------------ Fiber shard_queue0 (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b52f74  dfly::BufferedStreamerBase::NotifyWritten()
0x5644d9b7651f  dfly::journal::JournalSlice::AddLogRecord()
0x5644d9b7591a  dfly::journal::Journal::RecordEntry()
0x5644d9b80fac  dfly::Transaction::LogJournalOnShard()
0x5644d9b8111f  dfly::Transaction::LogAutoJournalOnShard()
0x5644d9b81248  dfly::Transaction::RunQuickie()
0x5644d9b813d6  dfly::Transaction::ScheduleUniqueShard()
0x5644d9b831d1  std::_Function_handler<>::_M_invoke()
0x5644da082b2c  util::fb2::FiberQueue::Run()
I20231119 14:32:39.001818 2564752 scheduler.cc:713] ------------ Fiber shard_periodic0 (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b530f4  dfly::BufferedStreamerBase::AwaitIfWritten()
0x5644d9b7651f  dfly::journal::JournalSlice::AddLogRecord()
0x5644d9b7591a  dfly::journal::Journal::RecordEntry()
0x5644d9b6fe3b  dfly::TriggerJournalWriteToSink()
0x5644d9a81c66  dfly::EngineShard::Heartbeat()
0x5644d9a82c38  dfly::EngineShard::RunPeriodic()
I20231119 14:32:39.003103 2564752 scheduler.cc:713] ------------ Fiber AcceptLoop (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c0da  util::fb2::UringSocket::Accept()
0x5644da056ef8  util::ListenerInterface::RunAcceptLoop()
I20231119 14:32:39.003762 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.005410 2564752 scheduler.cc:713] ------------ Fiber DflyConnection_repl_ctr (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.005416 2564752 scheduler.cc:713] ------------ Fiber DflyConnection_repl_flo (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.005421 2564752 scheduler.cc:713] ------------ Fiber journal_stream (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08e0a6  util::fb2::UringSocket::WriteSome()
0x5644da097f78  io::Sink::Write()
0x5644d9b52b28  dfly::BufferedStreamerBase::ConsumeIntoSink()
0x5644d9b8b026  dfly::JournalStreamer::WriterFb()
I20231119 14:32:39.006659 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006664 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006668 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006673 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006678 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006682 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006686 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006690 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006693 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006697 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006702 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006706 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006711 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006714 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006718 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006722 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006726 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006729 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006733 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006738 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006742 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006747 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006752 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006757 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006760 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006765 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006769 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006773 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006778 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006781 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006786 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006791 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006795 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006799 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006804 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006808 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006814 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006817 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006821 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.006997 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007011 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007016 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007020 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007025 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007030 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007035 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007038 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007042 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007047 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007052 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007061 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007066 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007069 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007074 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007078 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007083 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007087 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007091 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007097 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007100 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007105 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007110 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007114 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007119 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007123 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007128 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007134 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007138 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007143 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007146 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007150 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007155 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007160 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007165 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644da08c9fa  util::fb2::UringSocket::Recv()
0x5644d9c495d4  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.007169 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.009989 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010002 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010011 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010020 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010027 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010036 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010044 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010053 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010062 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010071 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010078 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010087 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010095 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010102 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010110 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010116 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010125 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010133 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010140 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010147 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010154 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010162 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010169 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010176 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010185 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d99ecb4c  dfly::(anonymous namespace)::SAdd()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c4965a  facade::Connection::IoLoop()
0x5644d9c499fd  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010193 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9b7c284  util::fb2::EventCount::wait()
0x5644d9b82d00  dfly::Transaction::ScheduleSingleHop()
0x5644d9a1ff44  dfly::StringFamily::Set()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c499a5  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.010900 2564752 scheduler.cc:713] ------------ Fiber DflyConnection (suspended) ------------
0x5644da04f6c0  util::fb2::detail::FiberInterface::SwitchTo()
0x5644da04c967  util::fb2::detail::Scheduler::Preempt()
0x5644d9a9a446  util::fb2::EventCount::await<>()
0x5644d9aa0f06  dfly::DebugCmd::Stacktrace()
0x5644d99a4b39  dfly::ServerFamily::Debug()
0x5644d9a902de  dfly::CommandId::Invoke()
0x5644d995fdc9  dfly::Service::InvokeCmd()
0x5644d99643db  dfly::Service::DispatchCommand()
0x5644d9c47810  facade::Connection::DispatchCommand()
0x5644d9c47a31  facade::Connection::ParseRedis()
0x5644d9c499a5  facade::Connection::ConnectionFlow()
0x5644d9c4a8b4  facade::Connection::HandleRequests()
0x5644da059656  util::ListenerInterface::RunSingleConnection()
I20231119 14:32:39.011852 2564752 scheduler.cc:713] ------------ Fiber  (active) ------------
0x5644da04e119  util::fb2::detail::Scheduler::ExecuteOnAllFiberStacks()
0x5644da04e295  util::fb2::detail::Scheduler::PrintAllFiberStackTraces()
chakaz commented 10 months ago

Oh, and continuing (via c in gdb I mean) the replication process immediately unfreezes everything

romange commented 10 months ago

yes,this is not surprising, give our we replicate data and wait for Sink::Write() to finish before proceeding in the shard queue.

chakaz commented 10 months ago

Perhaps not surprising :) But don't you think we should fix that?

dranikpg commented 10 months ago

We have only one alternative option - remove the replica if it lags behind too much

dranikpg commented 10 months ago

BufferedStreamerBase from io_utils has clearly defined limits on what gap we can keep before stopping

romange commented 10 months ago

Perhaps not surprising :) But don't you think we should fix that?

we should, could be something we work in Q1 :)