yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9.06k stars 1.09k forks source link

FATAL: operation's hybrid time too low #4032

Closed dechanggu closed 1 year ago

dechanggu commented 4 years ago

Jira Link: DB-4428 Log from YCSB threads:

Maximum time elapsed. Requesting stop for the workload. Stop requested for workload. Now Joining! java.lang.ArrayIndexOutOfBoundsException: Histogram recorded value cannot be negative. at org.HdrHistogram.AbstractHistogram.countsArrayIndex(AbstractHistogram.java:1944) at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:414) at org.HdrHistogram.AbstractHistogram.recordValue(AbstractHistogram.java:333) at org.HdrHistogram.Recorder.recordValue(Recorder.java:100) at site.ycsb.measurements.OneMeasurementHdrHistogram.measure(OneMeasurementHdrHistogram.java:101) at site.ycsb.measurements.Measurements.measure(Measurements.java:189) at site.ycsb.DBWrapper.measure(DBWrapper.java:178) at site.ycsb.DBWrapper.scan(DBWrapper.java:161) at site.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:795) at site.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:653) at site.ycsb.ClientThread.run(ClientThread.java:122) at java.base/java.lang.Thread.run(Thread.java:830)

dechanggu commented 4 years ago
Here is the yb-tserver.FATAL:
Log file created at: 2020/03/20 21:30:16
Running on machine: <hostname>
Application fingerprint: version 2.1.0.0 build 18 revision ead2d3186af17f0e9b94bf071444b6dfa5d11fd3 build_type RELEASE built at 22 Feb 2020 01:09:51 UTC
Running duration (h:mm:ss): 240:15:48
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0320 21:30:16.948240 18114 mvcc.cc:190] T 544cfce81b1b40ba990ca17ba66e918a P 889b1cf02c98482a958b646bde999451: New operation's hybrid time too low: { physical: 1584739816947634 }
  max_safe_time_returned_with_lease_={ safe_time: { physical: 1584739816947896 } source: kNow }
  *ht < max_safe_time_returned_with_lease_.safe_time=1
  static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_with_lease_.safe_time.ToUint64())=-1073152
  ht->PhysicalDiff(max_safe_time_returned_with_lease_.safe_time)=-262
  max_safe_time_returned_without_lease_={ safe_time: <min> source: kUnknown }
  *ht < max_safe_time_returned_without_lease_.safe_time=0
  static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_without_lease_.safe_time.ToUint64())=6491094290217508864
  ht->PhysicalDiff(max_safe_time_returned_without_lease_.safe_time)=1584739816947634
  max_safe_time_returned_for_follower_={ safe_time: <min> source: kUnknown }
  *ht < max_safe_time_returned_for_follower_.safe_time=0
  static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_for_follower_.safe_time.ToUint64())=6491094290217508864
  ht->PhysicalDiff(max_safe_time_returned_for_follower_.safe_time)=1584739816947634
  (SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown})={ safe_time: { physical: 1584739815102055 } source: kUnknown }
  *ht < (SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time=0
  static_cast<int64_t>(ht->ToUint64() - (SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time.ToUint64())=7559491584
  ht->PhysicalDiff((SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time)=1845579
  (SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown})={ safe_time: <min> source: kUnknown }
  *ht < (SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time=0
  static_cast<int64_t>(ht->ToUint64() - (SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time.ToUint64())=6491094290217508864
  ht->PhysicalDiff((SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time)=1584739816947634
  is_follower_side=1
  queue_.size()=0
  queue_=[]
  aborted=[]
ndeodhar commented 4 years ago

Seems to be similar to #4028

bmatican commented 4 years ago

@dechanggu There should also be a stack trace printed after the fatal message. Do you mind sharing that too?

mbautin commented 4 years ago

ead2d3186af17f0e9b94bf071444b6dfa5d11fd3 (2.1.0.0) does not have the commit 1932381cdc05905e4209449602123ebf15f6ca07 that could have fixed this issue (currently double-checking this).

rthallamko3 commented 1 year ago

DUP of https://github.com/yugabyte/yugabyte-db/issues/14225