search

MrCroxx / RunKV

[WIP] An experimental cloud-native distributed KV engine for OLTP workload.
MIT License
68 stars 7 forks source link

bug: abnormal fsync latency jitter #145

Closed MrCroxx closed 2 years ago

MrCroxx commented 2 years ago

Found some abnormal jitters in metrics:

image

After fio on the disk, fsync never reached second-level latency.

MrCroxx commented 2 years ago

Plus, there is no tight relationship between sycn size and abnormal sync duration:

abnormal sync duration: 55.965958ms size: 2523
abnormal sync duration: 50.050433ms size: 1095
abnormal sync duration: 73.884395ms size: 3419
abnormal sync duration: 50.422164ms size: 4566
abnormal sync duration: 57.817028ms size: 4308
abnormal sync duration: 77.529078ms size: 4686
abnormal sync duration: 92.60864ms size: 1701
abnormal sync duration: 171.006953ms size: 684
abnormal sync duration: 280.307331ms size: 1655
abnormal sync duration: 288.218533ms size: 4096
abnormal sync duration: 377.430867ms size: 2446
abnormal sync duration: 311.49896ms size: 1397
abnormal sync duration: 54.521834ms size: 1663
abnormal sync duration: 437.16799ms size: 2497
abnormal sync duration: 56.756441ms size: 682
abnormal sync duration: 61.188751ms size: 7657
abnormal sync duration: 293.984635ms size: 1828
abnormal sync duration: 71.521214ms size: 588
abnormal sync duration: 99.890314ms size: 6191
abnormal sync duration: 105.749283ms size: 3096
abnormal sync duration: 370.36068ms size: 1152
abnormal sync duration: 469.280632ms size: 4085
abnormal sync duration: 545.123856ms size: 4411
abnormal sync duration: 553.225657ms size: 1164
abnormal sync duration: 54.445309ms size: 4453
abnormal sync duration: 526.168279ms size: 878
abnormal sync duration: 459.310838ms size: 2281
abnormal sync duration: 71.620351ms size: 654
abnormal sync duration: 770.484198ms size: 3814
abnormal sync duration: 62.289419ms size: 1611
abnormal sync duration: 62.294725ms size: 2583
abnormal sync duration: 58.086945ms size: 7033
abnormal sync duration: 58.409421ms size: 5915
abnormal sync duration: 141.046483ms size: 465
abnormal sync duration: 962.8704ms size: 1186
abnormal sync duration: 67.774677ms size: 799
abnormal sync duration: 921.008574ms size: 2303
abnormal sync duration: 1.245822898s size: 481
abnormal sync duration: 66.604695ms size: 1925
abnormal sync duration: 56.685591ms size: 2228
abnormal sync duration: 51.218974ms size: 36
abnormal sync duration: 50.608881ms size: 57
abnormal sync duration: 1.1394668s size: 1413
abnormal sync duration: 55.858866ms size: 700
abnormal sync duration: 773.501331ms size: 1691
abnormal sync duration: 73.526351ms size: 4309
abnormal sync duration: 911.541348ms size: 6288
abnormal sync duration: 66.82163ms size: 2277
abnormal sync duration: 823.973216ms size: 2715
abnormal sync duration: 80.235226ms size: 2442
abnormal sync duration: 878.344511ms size: 1740
abnormal sync duration: 98.054495ms size: 3195
abnormal sync duration: 1.463387888s size: 1118
abnormal sync duration: 2.134725712s size: 5000
abnormal sync duration: 1.52571895s size: 2880
abnormal sync duration: 56.250808ms size: 1425
abnormal sync duration: 69.049605ms size: 4774
abnormal sync duration: 1.403227646s size: 1380
abnormal sync duration: 1.533101999s size: 3345
abnormal sync duration: 1.210609561s size: 1314
abnormal sync duration: 1.471951083s size: 1579
abnormal sync duration: 1.249606621s size: 1428
abnormal sync duration: 56.506391ms size: 1127
abnormal sync duration: 54.457477ms size: 1818
abnormal sync duration: 79.135706ms size: 3089
abnormal sync duration: 1.300945059s size: 2247
abnormal sync duration: 1.397921994s size: 1647
abnormal sync duration: 100.031688ms size: 2701
abnormal sync duration: 51.449189ms size: 8376
abnormal sync duration: 75.080663ms size: 2879
abnormal sync duration: 68.633048ms size: 4031
MrCroxx commented 2 years ago

Does sync frequency exceed the capacity of the disk?

MrCroxx commented 2 years ago

Seems bug on MacOS. Temporarily closed, reopen if needed.