[docdb] Check failed: _s.ok() Bad status: Service unavailable (yb/consensus/log.cc:336): WAL is shutting down (system error 108)

[rao-vasireddy]

Jira Link: DB-1698

F20200817 18:00:40 ../../src/yb/consensus/raft_consensus.cc:1855] Check failed: _s.ok() Bad status: Service unavailable (yb/consensus/log.cc:336): WAL is shutting down (system error 108) @ 0x7ffbc03cf92c yb::LogFatalHandlerSink::send() @ 0x7ffbbf5b5376 google::LogMessage::SendToLog() @ 0x7ffbbf5b27da google::LogMessage::Flush() @ 0x7ffbbf5b58a9 google::LogMessageFatal::~LogMessageFatal() @ 0x7ffbc88e3e60 yb::consensus::RaftConsensus::EnqueueWritesUnlocked() @ 0x7ffbc88f8c5e yb::consensus::RaftConsensus::UpdateReplica() @ 0x7ffbc88f9ac3 yb::consensus::RaftConsensus::Update() @ 0x7ffbc9569871 yb::tserver::ConsensusServiceImpl::UpdateConsensus() @ 0x7ffbc5e3ccba yb::consensus::ConsensusServiceIf::Handle() @ 0x7ffbc1c291d9 yb::rpc::ServicePoolImpl::Handle() @ 0x7ffbc1bcc074 yb::rpc::InboundCall::InboundCallTask::Run() @ 0x7ffbc1c34f28 yb::rpc::(anonymous namespace)::Worker::Execute() @ 0x7ffbc0464f3f yb::Thread::SuperviseThread() @ 0x7ffbbac88694 start_thread @ 0x7ffbba3c541d __clone @ (nil) (unknown)

[bmatican]

@rao-vasireddy any chance we had logs for this? or you know in what situation it might have happened? I suspect this was some tablet deletion race...

In any case though, does it make sense to actually crash here @mbautin ?

1966 yb::OpId RaftConsensus::EnqueueWritesUnlocked(const LeaderRequest& deduped_req,
1967                                               WriteEmpty write_empty) {
1968   // Now that we've triggered the prepares enqueue the operations to be written
1969   // to the WAL.
1970   if (PREDICT_TRUE(!deduped_req.messages.empty()) || write_empty) {
1971     // Trigger the log append asap, if fsync() is on this might take a while
1972     // and we can't reply until this is done.
1973     //
1974     // Since we've prepared, we need to be able to append (or we risk trying to apply
1975     // later something that wasn't logged). We crash if we can't.
1976     CHECK_OK(queue_->AppendOperations(
1977         deduped_req.messages, deduped_req.committed_op_id, state_->Clock().Now()));
1978   }
1979
1980   return !deduped_req.messages.empty() ?
1981       yb::OpId::FromPB(deduped_req.messages.back()->id()) : deduped_req.preceding_op_id;
1982 }

Based on the comment above in the main Update flow

1738   // 3 - We enqueue the writes to the WAL.
1739   //
1740   // We enqueue writes to the WAL, but only the operations that were successfully
1741   // enqueued for prepare (for the reasons introduced above). This means that even
1742   // if a prepare fails to enqueue, if any of the previous prepares were successfully
1743   // submitted they must be written to the WAL.
1744   // If writing to the WAL fails, we're in an inconsistent state and we crash. In this
1745   // case, no one will ever know of the operations we previously prepared so those are
1746   // inconsequential.

This makes sense, but I'm wondering if we should special case for log shutdown?

[bmatican]

As per @spolitov offline

I think that before starting update consensus, we should check whether tablet is in shutdown state. And reject update consensus in case of shutdown.
Also during shutdown we should not initiate shutdown while update consensus is running

cc @robertsami as another potential work item in the raft area

[bmatican]

Reducing priority as this should not be a crash loop, but rather a one off crash. Also, for this to happen, we'd have to have an inflight write, while a tablet peer is shutting down (so either drop table, or LB removing the tablet peer).

@rthallamko3 this might be a good task in Raft, if we want to onboard someone in that code path.