[DocDB][xCluster] Found New operation's hybrid time too low: FATAL log in target universe in xCluster transactional setup

[Arjun-yb]

Jira Link: DB-8219

Description

DB Version: 2.19.3.0-b87 It is transactional LRU setup

F20230928 08:04:42 ../../src/yb/tablet/mvcc.cc:395] T 12162ee857eb45b588a600742ea92155 P db3d356ed5654a7c875a07bb4e96c523: T 12162ee857eb45b588a600742ea92155 P db3d356ed5654a7c875a07bb4e96c523: Recent 32 MVCC operations:
1. UpdatePropagatedSafeTimeOnLeader { ht_lease: { time: { physical: 1695888278678557 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278678557 } }
2. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
3. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888278780963 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278780963 } }
4. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
5. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888278781006 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278781006 } }
6. UpdatePropagatedSafeTimeOnLeader { ht_lease: { time: { physical: 1695888278781037 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278781037 } }
7. AddLeaderPending { ht: { physical: 1695888278781236 } op_id: 26.4949604 }
8. UpdatePropagatedSafeTimeOnLeader { ht_lease: { time: { physical: 1695888278781488 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
9. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
10. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888278781790 } lease: { physical: 1695888280675602 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
11. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
12. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888278782638 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
13. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
14. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888281791998 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
15. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
16. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282204219 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
17. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
18. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282237001 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
19. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
20. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282746218 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
21. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
22. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282817515 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
23. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
24. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282837223 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
25. LastReplicatedHybridTime { last_replicated: { physical: 1695888278664206 } }
26. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282837296 } lease: { physical: 1695888280781007 } } safe_time: { physical: 1695888278781235 logical: 4095 } }
27. Replicated { ht: { physical: 1695888278781236 } op_id: 26.4949604 }
28. UpdatePropagatedSafeTimeOnLeader { ht_lease: { time: { physical: 1695888282837383 } lease: { physical: 1695888283792000 } } safe_time: { physical: 1695888282837383 } }
29. LastReplicatedHybridTime { last_replicated: { physical: 1695888278781236 } }
30. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282837396 } lease: { physical: 1695888283792000 } } safe_time: { physical: 1695888282837396 } }
31. LastReplicatedHybridTime { last_replicated: { physical: 1695888278781236 } }
32. SafeTime { min_allowed: <min> deadline: +inf ht_lease: { time: { physical: 1695888282837433 } lease: { physical: 1695888283792000 } } safe_time: { physical: 1695888282837433 } }
New operation's hybrid time too low: { physical: 1695888282808272 }, op id: 28.4949607
  max_safe_time_returned_with_lease_={ safe_time: { physical: 1695888282837433 } source: kNow }
  !!! ht <= max_safe_time_returned_with_lease_.safe_time=1
  static_cast<int64_t>(ht.ToUint64() - max_safe_time_returned_with_lease_.safe_time.ToUint64())=-119443456
  ht.PhysicalDiff(max_safe_time_returned_with_lease_.safe_time)=-29161

  max_safe_time_returned_without_lease_={ safe_time: <min> source: kUnknown }
  ht <= max_safe_time_returned_without_lease_.safe_time=0
  static_cast<int64_t>(ht.ToUint64() - max_safe_time_returned_without_lease_.safe_time.ToUint64())=6946358406382682112
  ht.PhysicalDiff(max_safe_time_returned_without_lease_.safe_time)=1695888282808272

  max_safe_time_returned_for_follower_={ safe_time: <min> source: kUnknown }
  ht <= max_safe_time_returned_for_follower_.safe_time=0
  static_cast<int64_t>(ht.ToUint64() - max_safe_time_returned_for_follower_.safe_time.ToUint64())=6946358406382682112
  ht.PhysicalDiff(max_safe_time_returned_for_follower_.safe_time)=1695888282808272

  last_replicated_={ physical: 1695888278781236 }
  ht <= last_replicated_=0
  static_cast<int64_t>(ht.ToUint64() - last_replicated_.ToUint64())=16494739456
  ht.PhysicalDiff(last_replicated_)=4027036

  last_ht_in_queue=<min>
  ht <= last_ht_in_queue=0
  static_cast<int64_t>(ht.ToUint64() - last_ht_in_queue.ToUint64())=6946358406382682112
  ht.PhysicalDiff(last_ht_in_queue)=1695888282808272

  propagated_safe_time_={ physical: 1695888282837383 }
  !!! ht <= propagated_safe_time_=1
  static_cast<int64_t>(ht.ToUint64() - propagated_safe_time_.ToUint64())=-119238656
  ht.PhysicalDiff(propagated_safe_time_)=-29111

  queue_.size()=0
  queue_=[]
    @     0x562382cf7567  google::LogMessage::SendToLog()
    @     0x562382cf84cd  google::LogMessage::Flush()
    @     0x562382cf8b49  google::LogMessageFatal::~LogMessageFatal()
    @     0x562383e4ede3  yb::tablet::MvccManager::AddPending()
    @     0x562383e52e7b  yb::tablet::MvccManager::AddFollowerPending()
    @     0x562383ee9869  yb::tablet::TabletPeer::NewOperationDriver()
    @     0x562383ee41bb  yb::tablet::TabletPeer::StartReplicaOperation()
    @     0x56238319fcb9  yb::consensus::RaftConsensus::StartReplicaOperationUnlocked()
    @     0x56238319abea  yb::consensus::RaftConsensus::UpdateReplica()
    @     0x56238317df09  yb::consensus::RaftConsensus::Update()
    @     0x56238414b8d7  yb::tserver::ConsensusServiceImpl::UpdateConsensus()
    @     0x562383206f71  std::__1::__function::__func<>::operator()()
    @     0x562383207baf  yb::consensus::ConsensusServiceIf::Handle()
    @     0x562383db0af0  yb::rpc::ServicePoolImpl::Handle()
    @     0x562383ceb54f  yb::rpc::InboundCall::InboundCallTask::Run()
    @     0x562383dbf6f3  yb::rpc::(anonymous namespace)::Worker::Execute()
    @     0x5623845208b1  yb::Thread::SuperviseThread()
    @     0x7f55e82d2694  start_thread
    @     0x7f55e85cf41d  __clone

Universe details are added in respective JIRA ticket - DB-8219

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[rthallamko3]

Lowering the priority as we do not have a repro.