Closed dechanggu closed 1 year ago
Here is the yb-tserver.FATAL:
Log file created at: 2020/03/20 21:30:16
Running on machine: <hostname>
Application fingerprint: version 2.1.0.0 build 18 revision ead2d3186af17f0e9b94bf071444b6dfa5d11fd3 build_type RELEASE built at 22 Feb 2020 01:09:51 UTC
Running duration (h:mm:ss): 240:15:48
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0320 21:30:16.948240 18114 mvcc.cc:190] T 544cfce81b1b40ba990ca17ba66e918a P 889b1cf02c98482a958b646bde999451: New operation's hybrid time too low: { physical: 1584739816947634 }
max_safe_time_returned_with_lease_={ safe_time: { physical: 1584739816947896 } source: kNow }
*ht < max_safe_time_returned_with_lease_.safe_time=1
static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_with_lease_.safe_time.ToUint64())=-1073152
ht->PhysicalDiff(max_safe_time_returned_with_lease_.safe_time)=-262
max_safe_time_returned_without_lease_={ safe_time: <min> source: kUnknown }
*ht < max_safe_time_returned_without_lease_.safe_time=0
static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_without_lease_.safe_time.ToUint64())=6491094290217508864
ht->PhysicalDiff(max_safe_time_returned_without_lease_.safe_time)=1584739816947634
max_safe_time_returned_for_follower_={ safe_time: <min> source: kUnknown }
*ht < max_safe_time_returned_for_follower_.safe_time=0
static_cast<int64_t>(ht->ToUint64() - max_safe_time_returned_for_follower_.safe_time.ToUint64())=6491094290217508864
ht->PhysicalDiff(max_safe_time_returned_for_follower_.safe_time)=1584739816947634
(SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown})={ safe_time: { physical: 1584739815102055 } source: kUnknown }
*ht < (SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time=0
static_cast<int64_t>(ht->ToUint64() - (SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time.ToUint64())=7559491584
ht->PhysicalDiff((SafeTimeWithSource{last_replicated_, SafeTimeSource::kUnknown}).safe_time)=1845579
(SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown})={ safe_time: <min> source: kUnknown }
*ht < (SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time=0
static_cast<int64_t>(ht->ToUint64() - (SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time.ToUint64())=6491094290217508864
ht->PhysicalDiff((SafeTimeWithSource{last_ht_in_queue, SafeTimeSource::kUnknown}).safe_time)=1584739816947634
is_follower_side=1
queue_.size()=0
queue_=[]
aborted=[]
Seems to be similar to #4028
@dechanggu There should also be a stack trace printed after the fatal message. Do you mind sharing that too?
ead2d3186af17f0e9b94bf071444b6dfa5d11fd3 (2.1.0.0) does not have the commit 1932381cdc05905e4209449602123ebf15f6ca07 that could have fixed this issue (currently double-checking this).
Jira Link: DB-4428 Log from YCSB threads:
Maximum time elapsed. Requesting stop for the workload. Stop requested for workload. Now Joining! java.lang.ArrayIndexOutOfBoundsException: Histogram recorded value cannot be negative. at org.HdrHistogram.AbstractHistogram.countsArrayIndex(AbstractHistogram.java:1944) at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:414) at org.HdrHistogram.AbstractHistogram.recordValue(AbstractHistogram.java:333) at org.HdrHistogram.Recorder.recordValue(Recorder.java:100) at site.ycsb.measurements.OneMeasurementHdrHistogram.measure(OneMeasurementHdrHistogram.java:101) at site.ycsb.measurements.Measurements.measure(Measurements.java:189) at site.ycsb.DBWrapper.measure(DBWrapper.java:178) at site.ycsb.DBWrapper.scan(DBWrapper.java:161) at site.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:795) at site.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:653) at site.ycsb.ClientThread.run(ClientThread.java:122) at java.base/java.lang.Thread.run(Thread.java:830)