search

oxidecomputer / omicron

Omicron: Oxide control plane
Mozilla Public License 2.0
244 stars 36 forks source link

CRDB reported clock sync error after sled-agent reported time was synchronised #3831

Closed askfongjojo closed 1 year ago

askfongjojo commented 1 year ago

We saw this issue with 2 of the 5 cockroachdb nodes today on rack2 (gc12 and gc25) after running in-place update following the prepare-update procedures (as opposed to running the clean-slate script).

Here is the timeline of key events on sled 12, BRM42220057. At 23:33:27.463Z, time sync was completed: 

23:33:24.610Z WARN SledAgent (BootstrapAgent): Time not yet synchronised (retrying in 989.573484ms)
    error = "No sync TimeSync { sync: false, skew: 0.0, correction: 0.056633085 }"
    file = sled-agent/src/services.rs:643
23:33:25.620Z WARN SledAgent (BootstrapAgent): Time not yet synchronised (retrying in 1.241705837s)
    error = "No sync TimeSync { sync: false, skew: 0.0, correction: 0.053904276 }"
    file = sled-agent/src/services.rs:643
23:33:27.409Z INFO SledAgent (BootstrapAgent): Setting boot time to 1691451207.409079368s
    file = sled-agent/src/services.rs:2774
23:33:27.423Z WARN SledAgent (BootstrapAgent): Updating /pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/utmpx failed: Comm
and [/usr/platform/oxide/bin/tmpx 1691451207 /pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/utmpx] executed and failed wi
th status: exit status: 1  stdout:   stderr: tmpx: Invalid path: '/pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/utmpx' (
see utmpxname(3C))
    file = sled-agent/src/services.rs:2791
23:33:27.436Z WARN SledAgent (BootstrapAgent): Updating /pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/wtmpx failed: Comm
and [/usr/platform/oxide/bin/tmpx 1691451207 /pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/wtmpx] executed and failed wi
th status: exit status: 1  stdout:   stderr: tmpx: Invalid path: '/pool/ext/114c2d91-5377-4b38-8908-d9faef49b223/crypt/zone/oxz_ntp_47b1265d-060f-45bd-8b7f-a59cf3736fc7/root/var/adm/wtmpx' (
see utmpxname(3C))
    file = sled-agent/src/services.rs:2791
23:33:27.450Z INFO SledAgent (BootstrapAgent): Updated /var/adm/utmpx
    file = sled-agent/src/services.rs:2794
23:33:27.463Z INFO SledAgent (BootstrapAgent): Updated /var/adm/wtmpx
    file = sled-agent/src/services.rs:2794
23:33:27.463Z INFO SledAgent (BootstrapAgent): Time is synchronized

CRDB zone was provisioned afterwards:

BRM42220057 # grep 'Omicron zone: oxz_cockroachdb_' /pool/ext/eb8257c9-d723-46f0-b537-18c73f0959c0/crypt/debug/global/oxide-sled-agent:default.log.1691462308 | looker
23:33:28.063Z INFO SledAgent (BootstrapAgent): Configuring new Omicron zone: oxz_cockroachdb_ddda4b9c-c07b-4acb-84b5-d44cc1b1a109
    file = illumos-utils/src/zone.rs:314
23:33:28.206Z INFO SledAgent (BootstrapAgent): Installing Omicron zone: oxz_cockroachdb_ddda4b9c-c07b-4acb-84b5-d44cc1b1a109
    file = illumos-utils/src/zone.rs:353

The cockroachdb SMF service however went into maintenance:

[ Aug  7 23:34:14 Method "start" exited with status 0. ]
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
* 
* This mode is intended for non-production testing only.
* 
* In this mode:
* - Your cluster is open to any client that can access fd00:1122:3344:102::3.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
* 
* - https://go.crdb.dev/issue-v/53404/v22.1
* - https://www.cockroachlabs.com/docs/v22.1/secure-a-cluster.html
*
*
* ERROR: Queued as error 6f7a696b2f0a44d1bb827cc70f5d4cf4
*
[ Aug  7 23:34:17 Stopping because all processes in service exited. ]
[ Aug  7 23:34:17 Executing stop method (:kill). ]
[ Aug  7 23:34:17 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]

There were several retries:

BRM42220057 # grep kill /pool/ext/eb8257c9-d723-46f0-b537-18c73f0959c0/crypt/debug/oxz_cockroachdb_ddda4b9c-c07b-4acb-84b5-d44cc1b1a109/oxide-cockroachdb\:default.log.1691451280 
[ Aug  7 23:34:17 Executing stop method (:kill). ]
[ Aug  7 23:34:24 Executing stop method (:kill). ]
[ Aug  7 23:34:32 Executing stop method (:kill). ]
[ Aug  7 23:34:40 Executing stop method (:kill). ]

In the CRDB logs, we saw that it complained about a time drift of close to 500ms in each of the retries:

root@oxz_cockroachdb_ddda4b9c-c07b-4acb-84b5-d44cc1b1a109:~# cd /data/logs
root@oxz_cockroachdb_ddda4b9c-c07b-4acb-84b5-d44cc1b1a109:/data/logs# grep clock cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T*.log | grep -v /ws/gc/cockroach
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_14Z.010878.log:I230807 23:34:14.745813 161 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_14Z.010878.log:I230807 23:34:14.749308 161 1@server/clock_monotonicity.go:140 ⋮ [n4] 19  Sleeping till wall time 1691451254749305328 to catches up to 1691451255245755527 to ensure monotonicity. Delta: 496.450199ms
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_14Z.010878.log:F230807 23:34:17.522098 520 1@server/server.go:246 ⋮ [n4] 69  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_14Z.010878.log:F230807 23:34:17.522098 520 1@server/server.go:246 ⋮ [n4] 69 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_23Z.010904.log:I230807 23:34:23.667895 167 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_23Z.010904.log:I230807 23:34:23.670078 167 1@server/clock_monotonicity.go:140 ⋮ [n4] 19  Sleeping till wall time 1691451263670072636 to catches up to 1691451264167873529 to ensure monotonicity. Delta: 497.800893ms
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_23Z.010904.log:F230807 23:34:24.320524 904 1@server/server.go:246 ⋮ [n4] 46  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_23Z.010904.log:F230807 23:34:24.320524 904 1@server/server.go:246 ⋮ [n4] 46 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_30Z.010933.log:I230807 23:34:30.240086 200 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_30Z.010933.log:I230807 23:34:30.243118 200 1@server/clock_monotonicity.go:140 ⋮ [n4] 19  Sleeping till wall time 1691451270243112177 to catches up to 1691451270740056611 to ensure monotonicity. Delta: 496.944434ms
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_30Z.010933.log:F230807 23:34:32.783051 1157 1@server/server.go:246 ⋮ [n4] 60  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_30Z.010933.log:F230807 23:34:32.783051 1157 1@server/server.go:246 ⋮ [n4] 60 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_38Z.010967.log:I230807 23:34:38.744358 228 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_38Z.010967.log:I230807 23:34:38.746730 228 1@server/clock_monotonicity.go:140 ⋮ [n4] 19  Sleeping till wall time 1691451278746721043 to catches up to 1691451279244340433 to ensure monotonicity. Delta: 497.61939ms
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_38Z.010967.log:F230807 23:34:40.913372 367 1@server/server.go:246 ⋮ [n4] 58  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdbddda4b9c-c07b-4acb-84b5-d44cc1b1a109.root.2023-08-07T23_34_38Z.010967.log:F230807 23:34:40.913372 367 1@server/server.go:246 ⋮ [n4] 58 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
citrus-it commented 1 year ago

Here is the tracking log from BRM44220010's NTP zone. One interesting thing here is that the initial time synchronisation occurs with stratum 11, which is one more than the stratum advertised by the boundary servers when they are not yet in sync with the upstream servers - that is they are running free. That's expected, we still want the rack to maintain consistent time across sleds even when upstream is lost. The stratum then changes to 4 which means that the boundary servers are in sync with catacomb (stratum 2 at present). At that point, the time offset changes from 0.000002448 to -0.001149 and quickly gets back to the order of e-5.

This all looks ok to me, once we synchronised I don't see the offset going anywhere near to 500ms.

===================================================================================================================================
   Date (UTC) Time     IP Address        St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:03:11 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:04:41 fd00:1122:3344:10a::e 11      0.000 1000000.000 -1.155e+09 N  1  2.975e-02  0.000e+00  8.865e-05  1.163e+09  1.500e+00
2023-08-07 23:35:19 fd00:1122:3344:10a::e 11      0.000 1000000.000 -4.572e-01 N  1  2.587e-05  0.000e+00  9.129e-05  2.391e-01  1.155e+09
2023-08-07 23:35:20 fd00:1122:3344:10a::e 11  -2618.114    699.920 -2.651e-03 N  1  5.929e-05  4.545e-01  1.526e-04  4.124e-04  2.211e+00
2023-08-07 23:35:21 fd00:1122:3344:10a::e 11  -2571.044    394.234  8.691e-05 N  1  5.587e-05  4.544e-01  1.642e-04  4.065e-04  4.584e-01
2023-08-07 23:35:22 fd00:1122:3344:10a::e 11  -2571.044    394.234  1.437e-03 N  1  6.623e-04  4.516e-01  1.894e-04  8.381e-04  4.552e-01
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11  -2571.044    394.234  2.228e-03 N  1  1.037e-03  4.474e-01  1.907e-04  1.573e-03  4.521e-01
2023-08-07 23:35:24 fd00:1122:3344:10a::e 11  -2571.044    394.234  3.609e-03 N  1  8.591e-04  4.425e-01  1.429e-04  2.166e-03  4.484e-01
2023-08-07 23:35:25 fd00:1122:3344:10a::e 11  -2571.044    394.234  2.623e-03 N  1  5.792e-04  4.361e-01  1.797e-04  2.646e-03  4.432e-01
2023-08-07 23:35:26 fd00:1122:3344:10a::e 11      1.116    161.752  3.144e-03 N  1  1.204e-05  4.307e-01  2.019e-04  6.277e-05  4.388e-01
2023-08-07 23:35:27 fd00:1122:3344:10a::e 11      1.116    161.752  1.633e-03 N  1  5.001e-04  4.249e-01  1.542e-04  5.272e-04  4.279e-01
2023-08-07 23:35:28 fd00:1122:3344:10a::e 11      1.116    161.752  5.958e-04 N  1  3.546e-04  4.205e-01  1.270e-04  5.176e-04  4.245e-01
2023-08-07 23:35:29 fd00:1122:3344:10a::e 11     41.520    361.710  3.367e-04 N  1  2.831e-04  4.172e-01  1.333e-04  1.033e-03  4.212e-01
2023-08-07 23:35:30 fd00:1122:3344:10a::e 11     41.520    361.710  9.742e-05 N  1  3.790e-04  4.141e-01  1.470e-04  1.010e-03  4.188e-01
2023-08-07 23:35:31 fd00:1122:3344:10a::e 11     41.520    361.710 -3.518e-04 N  1  2.959e-04  4.112e-01  1.145e-04  3.152e-04  4.157e-01
2023-08-07 23:35:32 fd00:1122:3344:10a::e 11      6.427     44.172 -1.696e-04 N  1  1.337e-05  4.088e-01  1.508e-04  5.108e-05  4.124e-01
2023-08-07 23:35:33 fd00:1122:3344:10a::e 11      2.338     34.321 -1.944e-05 N  1  1.539e-05  4.063e-01  1.391e-04  3.827e-05  4.092e-01
2023-08-07 23:35:34 fd00:1122:3344:10a::e 11     -1.880     48.903 -2.275e-05 N  1  1.984e-05  4.036e-01  1.455e-04  4.750e-05  4.064e-01
2023-08-07 23:35:35 fd00:1122:3344:10a::e 11    -12.724     31.528 -2.355e-05 N  1  1.040e-05  4.009e-01  1.940e-04  2.164e-05  4.038e-01
2023-08-07 23:35:36 fd00:1122:3344:10a::e 11    -12.638     12.857  4.163e-06 N  1  6.780e-06  3.981e-01  1.262e-04  1.052e-05  4.010e-01
2023-08-07 23:35:37 fd00:1122:3344:10a::e 11    -11.511     10.618  7.137e-06 N  1  7.340e-06  3.954e-01  1.262e-04  1.941e-05  3.982e-01
2023-08-07 23:35:38 fd00:1122:3344:10a::e 11    -10.597      7.936  5.241e-06 N  1  6.767e-06  3.927e-01  1.392e-04  1.114e-05  3.955e-01
2023-08-07 23:35:39 fd00:1122:3344:10a::e 11     -9.842      6.450  5.166e-06 N  1  6.644e-06  3.900e-01  1.436e-04  1.197e-05  3.928e-01
2023-08-07 23:35:40 fd00:1122:3344:10a::e 11     -9.099      5.852  5.923e-06 N  1  7.106e-06  3.872e-01  1.535e-04  1.360e-05  3.900e-01
2023-08-07 23:35:41 fd00:1122:3344:10a::e 11     -5.057      7.747  2.097e-05 N  1  4.934e-06  3.845e-01  1.528e-04  5.184e-06  3.873e-01
2023-08-07 23:35:42 fd00:1122:3344:10a::e 11     -3.513      5.011  3.292e-06 N  1  4.664e-06  3.817e-01  1.440e-04  9.442e-06  3.845e-01
2023-08-07 23:35:43 fd00:1122:3344:10a::e 11     -2.698      4.775  6.367e-06 N  1  5.548e-06  3.789e-01  1.440e-04  1.158e-05  3.818e-01
2023-08-07 23:35:44 fd00:1122:3344:10a::e 11     -2.905      3.186 -8.387e-07 N  1  3.934e-06  3.762e-01  1.121e-04  4.510e-06  3.790e-01
2023-08-07 23:35:45 fd00:1122:3344:10a::e 11     -2.657      3.232  2.917e-06 N  1  4.785e-06  3.735e-01  1.121e-04  1.005e-05  3.763e-01
2023-08-07 23:35:46 fd00:1122:3344:10a::e 11     -2.653      2.711 -5.897e-07 N  1  4.516e-06  3.707e-01  1.121e-04  1.297e-05  3.735e-01
2023-08-07 23:35:47 fd00:1122:3344:10a::e 11     -2.471      2.624  2.253e-06 N  1  4.840e-06  3.680e-01  1.121e-04  1.323e-05  3.708e-01
2023-08-07 23:35:48 fd00:1122:3344:10a::e 11     -2.218      2.407  2.724e-06 N  1  4.712e-06  3.653e-01  1.121e-04  1.484e-05  3.681e-01
===================================================================================================================================
   Date (UTC) Time     IP Address        St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:35:49 fd00:1122:3344:10a::e 11     -2.563      2.108 -5.160e-06 N  1  4.147e-06  3.625e-01  1.154e-04  5.880e-06  3.653e-01
2023-08-07 23:35:50 fd00:1122:3344:10a::e 11     -2.486      1.891  1.859e-06 N  1  4.355e-06  3.598e-01  1.154e-04  6.949e-06  3.626e-01
2023-08-07 23:35:51 fd00:1122:3344:10a::e 11     -2.358      1.896  1.859e-06 N  1  4.806e-06  3.570e-01  1.154e-04  1.103e-05  3.599e-01
2023-08-07 23:35:53 fd00:1122:3344:10a::e 11     -2.101      2.067  4.364e-06 N  1  6.028e-06  3.514e-01  1.154e-04  1.696e-05  3.571e-01
2023-08-07 23:35:55 fd00:1122:3344:10a::e 11     -1.900      1.983  2.803e-06 N  1  6.609e-06  3.460e-01  1.154e-04  2.080e-05  3.515e-01
2023-08-07 23:35:57 fd00:1122:3344:10a::e 11     -1.620      2.015  5.478e-06 N  1  7.506e-06  3.405e-01  1.154e-04  2.491e-05  3.461e-01
2023-08-07 23:35:59 fd00:1122:3344:10a::e 11     -1.279      2.073  7.027e-06 N  1  8.526e-06  3.350e-01  1.154e-04  3.045e-05  3.406e-01
2023-08-07 23:36:01 fd00:1122:3344:10a::e 11     -1.368      1.675 -4.123e-06 N  1  7.790e-06  3.295e-01  1.467e-04  9.840e-06  3.351e-01
2023-08-07 23:36:05 fd00:1122:3344:10a::e 11     -1.293      1.532  2.752e-06 N  1  8.696e-06  3.187e-01  1.467e-04  1.634e-05  3.296e-01
2023-08-07 23:36:10 fd00:1122:3344:10a::e 11     -1.290      2.206  1.386e-03 N  1  5.721e-04  3.077e-01  2.181e-04  6.523e-04  3.188e-01
2023-08-07 23:36:11 fd00:1122:3344:10a::e 11     -1.274      4.045  6.950e-04 N  1  4.144e-04  3.036e-01  1.838e-04  4.534e-04  3.072e-01
2023-08-07 23:36:12 fd00:1122:3344:10a::e 11     -1.168      9.101  2.573e-04 N  1  2.491e-04  3.001e-01  1.320e-04  2.746e-04  3.036e-01
2023-08-07 23:36:13 fd00:1122:3344:10a::e 11     -0.439     23.057  1.902e-04 N  1  1.940e-04  2.972e-01  1.717e-04  4.371e-04  3.005e-01
2023-08-07 23:36:14 fd00:1122:3344:10a::e 11      0.257     38.023  1.405e-04 N  1  2.443e-04  2.942e-01  1.690e-04  5.006e-04  2.977e-01
2023-08-07 23:36:15 fd00:1122:3344:10a::e 11     -0.646     46.852 -3.793e-04 N  1  1.715e-05  2.913e-01  2.046e-04  4.155e-05  2.950e-01
2023-08-07 23:36:16 fd00:1122:3344:10a::e 11     -1.045     29.784 -1.567e-07 N  1  1.449e-05  2.890e-01  1.862e-04  2.242e-05  2.919e-01
2023-08-07 23:36:17 fd00:1122:3344:10a::e 11     -1.301     18.969 -1.079e-06 N  1  1.273e-05  2.862e-01  1.936e-04  2.695e-05  2.891e-01
2023-08-07 23:36:18 fd00:1122:3344:10a::e 11     -8.586     27.049 -2.276e-05 N  1  7.566e-06  2.835e-01  1.109e-04  9.114e-06  2.864e-01
2023-08-07 23:36:19 fd00:1122:3344:10a::e 11     -1.953     22.090  5.606e-06 N  1  5.614e-06  2.808e-01  1.109e-04  4.648e-05  2.836e-01
2023-08-07 23:36:20 fd00:1122:3344:10a::e 11     -1.437     11.930 -8.149e-07 N  1  5.060e-06  2.780e-01  1.521e-04  1.586e-05  2.809e-01
2023-08-07 23:36:21 fd00:1122:3344:10a::e 11     -0.513     11.755  2.448e-06 N  1  5.316e-06  2.753e-01  1.521e-04  1.979e-05  2.781e-01
2023-08-07 23:36:22 fd00:1122:3344:10a::e  4     -0.513     11.755 -1.149e-03 N  1  6.605e-04  2.726e-01  1.925e-04  2.424e-03  2.754e-01
2023-08-07 23:36:23 fd00:1122:3344:10a::e  4     -0.513     11.755 -9.564e-04 N  1  7.037e-04  2.710e-01  1.925e-04  4.444e-03  2.766e-01
2023-08-07 23:36:24 fd00:1122:3344:10a::e  4     -0.513     11.755 -8.261e-04 N  1  5.178e-04  2.692e-01  1.925e-04  4.538e-03  2.770e-01
2023-08-07 23:36:25 fd00:1122:3344:10a::e  4     -0.911     27.938 -2.151e-04 N  1  3.556e-04  2.673e-01  1.925e-04  3.762e-03  2.753e-01
2023-08-07 23:36:26 fd00:1122:3344:10a::e  4     -0.911     27.938 -4.357e-05 N  1  3.954e-04  2.648e-01  1.925e-04  8.309e-03  2.720e-01
2023-08-07 23:36:27 fd00:1122:3344:10a::e  4    -38.989     63.526  2.525e-04 N  1  9.038e-06  2.621e-01  5.825e-02  1.545e-03  2.737e-01
2023-08-07 23:36:28 fd00:1122:3344:10a::e  4    -66.433     34.592 -3.550e-05 N  1  7.291e-06  2.591e-01  5.825e-02  1.590e-03  2.926e-01
2023-08-07 23:36:29 fd00:1122:3344:10a::e  4    -66.179     20.716  5.070e-06 N  1  8.812e-06  2.564e-01  5.827e-02  1.544e-03  2.899e-01
2023-08-07 23:36:30 fd00:1122:3344:10a::e  4    -64.976     27.699  9.065e-06 N  1  1.144e-05  2.537e-01  5.825e-02  1.657e-03  2.871e-01
2023-08-07 23:36:31 fd00:1122:3344:10a::e  4    -60.815     42.762  3.721e-05 N  1  2.073e-05  2.509e-01  5.828e-02  1.735e-03  2.845e-01
2023-08-07 23:36:32 fd00:1122:3344:10a::e  4    -54.426     30.130  1.718e-05 N  1  1.588e-05  2.482e-01  5.825e-02  1.688e-03  2.818e-01
===================================================================================================================================
   Date (UTC) Time     IP Address        St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:36:33 fd00:1122:3344:10a::e  4    -54.312     16.837 -3.010e-06 N  1  1.239e-05  2.454e-01  5.827e-02  1.620e-03  2.790e-01
2023-08-07 23:36:34 fd00:1122:3344:10a::e  4    -52.980     14.740  1.017e-05 N  1  1.361e-05  2.427e-01  5.827e-02  1.624e-03  2.762e-01
2023-08-07 23:36:35 fd00:1122:3344:10a::e  4    -52.382     10.540  2.972e-06 N  1  1.152e-05  2.399e-01  5.827e-02  1.604e-03  2.735e-01
2023-08-07 23:36:36 fd00:1122:3344:10a::e  4    -50.810      8.762  1.360e-05 N  1  1.016e-05  2.372e-01  5.827e-02  1.591e-03  2.707e-01
2023-08-07 23:36:37 fd00:1122:3344:10a::e  4    -49.737      7.403  8.516e-06 N  1  9.799e-06  2.345e-01  5.827e-02  1.580e-03  2.680e-01
2023-08-07 23:36:38 fd00:1122:3344:10a::e  4    -50.144      5.834 -5.943e-06 N  1  8.627e-06  2.317e-01  5.827e-02  1.574e-03  2.652e-01
2023-08-07 23:36:39 fd00:1122:3344:10a::e  4    -51.255      5.576 -1.784e-06 N  1  8.084e-06  2.290e-01  5.827e-02  1.572e-03  2.624e-01
2023-08-07 23:36:40 fd00:1122:3344:10a::e  4    -51.086      4.486  4.140e-06 N  1  7.428e-06  2.262e-01  5.827e-02  1.563e-03  2.597e-01
2023-08-07 23:36:41 fd00:1122:3344:10a::e  4    -51.665      4.219 -1.335e-06 N  1  7.227e-06  2.235e-01  5.827e-02  1.562e-03  2.569e-01
2023-08-07 23:36:42 fd00:1122:3344:10a::e  4    -52.456      4.098  2.185e-06 N  1  7.278e-06  2.208e-01  5.827e-02  1.564e-03  2.542e-01
2023-08-07 23:36:43 fd00:1122:3344:10a::e  4    -52.551      3.437  6.649e-07 N  1  7.104e-06  2.180e-01  5.827e-02  1.562e-03  2.515e-01
2023-08-07 23:36:44 fd00:1122:3344:10a::e  4    -52.624      3.048 -8.475e-07 N  1  6.797e-06  2.153e-01  5.827e-02  1.558e-03  2.487e-01
2023-08-07 23:36:46 fd00:1122:3344:10a::e  4    -52.590      2.524  2.728e-06 N  1  6.238e-06  2.098e-01  5.827e-02  1.551e-03  2.460e-01
2023-08-07 23:36:48 fd00:1122:3344:10a::e  4    -52.529      2.305  9.788e-07 N  1  6.624e-06  2.044e-01  5.827e-02  1.554e-03  2.405e-01
2023-08-07 23:36:50 fd00:1122:3344:10a::e  4    -52.305      2.314  4.201e-06 N  1  7.399e-06  1.989e-01  5.827e-02  1.556e-03  2.351e-01
2023-08-07 23:36:52 fd00:1122:3344:10a::e  4    -51.829      2.513  9.408e-06 N  1  8.550e-06  1.934e-01  5.827e-02  1.564e-03  2.296e-01
2023-08-07 23:36:54 fd00:1122:3344:10a::e  4    -51.588      2.200  2.921e-06 N  1  8.508e-06  1.879e-01  5.827e-02  1.563e-03  2.241e-01
2023-08-07 23:36:58 fd00:1122:3344:10a::e  4    -51.410      1.964  3.761e-06 N  1  9.351e-06  1.770e-01  5.827e-02  1.564e-03  2.186e-01
2023-08-07 23:37:02 fd00:1122:3344:10a::e  4    -50.801      3.680  2.076e-05 N  1  9.250e-06  1.660e-01  5.823e-02  2.170e-03  2.077e-01
2023-08-07 23:37:06 fd00:1122:3344:10a::e  4    -50.330      3.423 -1.839e-05 N  1  8.634e-06  1.549e-01  5.823e-02  2.154e-03  1.973e-01
2023-08-07 23:37:10 fd00:1122:3344:10a::e  4    -50.382      2.598 -3.147e-06 N  1  8.771e-06  1.440e-01  5.823e-02  2.150e-03  1.863e-01
2023-08-07 23:37:14 fd00:1122:3344:10a::e  4    -50.609      2.051 -6.194e-06 N  1  8.050e-06  1.331e-01  5.823e-02  2.142e-03  1.753e-01
2023-08-07 23:37:18 fd00:1122:3344:10a::e  4    -50.789      1.613 -4.601e-06 N  1  7.444e-06  1.222e-01  5.823e-02  2.135e-03  1.644e-01
2023-08-07 23:37:22 fd00:1122:3344:10a::e  4    -50.690      1.202  5.332e-06 N  1  6.379e-06  1.113e-01  5.823e-02  2.124e-03  1.535e-01
2023-08-07 23:37:26 fd00:1122:3344:10a::e  4    -50.697      1.621 -3.045e-04 N  1  1.481e-04  1.004e-01  5.816e-02  1.533e-03  1.425e-01
2023-08-07 23:37:27 fd00:1122:3344:10a::e  4    -56.522      2.113 -3.765e-04 N  2  3.449e-04  9.800e-02  5.817e-02  1.554e-03  1.314e-01
2023-08-07 23:37:28 fd00:1122:3344:10a::e  4    -56.051      2.880 -2.574e-05 N  2  2.664e-04  9.566e-02  5.819e-02  1.569e-03  1.290e-01
2023-08-07 23:37:29 fd00:1122:3344:10a::e  4    -55.865      2.891 -9.053e-05 N  2  1.500e-04  9.295e-02  5.823e-02  1.568e-03  1.264e-01
2023-08-07 23:37:30 fd00:1122:3344:10a::e  4    -55.849      2.798 -5.199e-05 N  2  8.662e-05  9.029e-02  5.818e-02  1.556e-03  1.237e-01
2023-08-07 23:37:31 fd00:1122:3344:10a::e  4    -55.855      2.760  1.011e-04 N  2  1.859e-04  8.761e-02  5.818e-02  1.489e-03  1.210e-01
2023-08-07 23:37:32 fd00:1122:3344:10a::e  4    -54.951      6.050  5.372e-05 N  2  1.434e-04  8.479e-02  5.813e-02  1.494e-03  1.181e-01
2023-08-07 23:37:33 fd00:1122:3344:10a::e  4    -52.893      9.807  3.633e-07 N  2  1.366e-04  8.200e-02  5.816e-02  1.489e-03  1.153e-01
citrus-it commented 1 year ago

The measurements log shows a similar story, although there was a period between 2023-08-07 23:35:20 and 2023-08-07 23:36:22 where one of the boundary servers was synced but not the other. Still well within a 500ms offset though.

========================================================================================================================================
   Date (UTC) Time     IP Address   L St 123 567 ABCD  LP RP Score    Offset  Peer del. Peer disp.  Root del. Root disp. Refid     MTxRx
========================================================================================================================================
1986-12-28 00:04:27 fd00:1122:3344:10a::e N 10 111 111 1101   1  1 0.99  1.155e+09  1.727e-04  5.989e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
1986-12-28 00:04:29 fd00:1122:3344:10a::e N 10 111 111 1111   2  1 0.20  1.155e+09  9.353e-05  5.985e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
1986-12-28 00:04:30 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.81  1.155e+09  2.097e-04  6.008e-08  5.807e-02  2.487e-03 AC140005 4B D K
1986-12-28 00:04:33 fd00:1122:3344:10a::e N 10 111 111 1111   2  2 0.42  1.155e+09  1.150e-04  5.985e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
1986-12-28 00:04:34 fd00:1122:3344:107::e N  3 111 111 1111   3  2 0.11  1.155e+09  1.320e-04  5.997e-08  5.807e-02  2.579e-03 AC140005 4B D K
1986-12-28 00:04:37 fd00:1122:3344:10a::e N 10 111 111 1111   2  2 0.65  1.155e+09  1.353e-04  5.987e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
1986-12-28 00:04:41 fd00:1122:3344:10a::e N 10 111 111 1111   0  2 0.00  1.155e+09  8.865e-05  5.984e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:19 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  4.572e-01  9.129e-05  2.843e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:20 fd00:1122:3344:107::e N  3 111 111 1111   0  3 0.00 -1.065e-04  1.386e-04  5.995e-08  5.807e-02  1.221e-03 AC140005 4B D K
2023-08-07 23:35:20 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  2.650e-03  1.526e-04  4.164e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:21 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -2.387e-03  1.931e-04  5.362e-07  5.807e-02  1.236e-03 AC140005 4B D K
2023-08-07 23:35:21 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -1.073e-04  1.642e-04  1.870e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:22 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -4.556e-03  1.955e-04  2.889e-07  5.807e-02  1.251e-03 AC140005 4B D K
2023-08-07 23:35:22 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -2.653e-03  1.894e-04  1.332e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:23 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -5.669e-03  1.773e-04  5.001e-07  5.807e-02  1.251e-03 AC140005 4B D K
2023-08-07 23:35:23 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -3.791e-03  1.907e-04  4.929e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:24 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -5.933e-03  1.960e-04  5.377e-07  5.807e-02  1.266e-03 AC140005 4B D K
2023-08-07 23:35:24 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -4.160e-03  1.429e-04  3.845e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:25 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -4.873e-03  1.879e-04  5.331e-07  5.807e-02  1.282e-03 AC140005 4B D K
2023-08-07 23:35:25 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -3.152e-03  1.797e-04  4.595e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:26 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -4.805e-03  2.118e-04  5.361e-07  5.807e-02  1.297e-03 AC140005 4B D K
2023-08-07 23:35:26 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -3.125e-03  2.019e-04  4.191e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:27 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -4.238e-03  1.718e-04  1.061e-07  5.807e-02  1.312e-03 AC140005 4B D K
2023-08-07 23:35:27 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -2.239e-03  1.542e-04  6.561e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:28 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -2.576e-03  2.194e-04  4.123e-07  5.807e-02  1.312e-03 AC140005 4B D K
2023-08-07 23:35:28 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -5.949e-04  1.270e-04  2.849e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:29 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.922e-03  2.366e-04  4.404e-07  5.807e-02  1.328e-03 AC140005 4B D K
2023-08-07 23:35:29 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  7.565e-06  1.333e-04  2.326e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:30 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.506e-03  1.901e-04  4.189e-07  5.807e-02  1.343e-03 AC140005 4B D K
2023-08-07 23:35:30 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  3.376e-04  1.470e-04  2.179e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:31 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.314e-03  1.771e-04  4.135e-07  5.807e-02  1.358e-03 AC140005 4B D K
2023-08-07 23:35:31 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  4.867e-04  1.145e-04  2.607e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
========================================================================================================================================
   Date (UTC) Time     IP Address   L St 123 567 ABCD  LP RP Score    Offset  Peer del. Peer disp.  Root del. Root disp. Refid     MTxRx
========================================================================================================================================
2023-08-07 23:35:32 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.569e-03  2.085e-04  3.772e-07  5.807e-02  1.373e-03 AC140005 4B D K
2023-08-07 23:35:32 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  1.905e-04  1.508e-04  2.479e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:33 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.638e-03  1.509e-04  6.259e-08  5.807e-02  1.373e-03 AC140005 4B D K
2023-08-07 23:35:33 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  5.921e-05  1.391e-04  6.625e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:34 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.608e-03  1.796e-04  6.652e-08  5.807e-02  1.389e-03 AC140005 4B D K
2023-08-07 23:35:34 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  5.442e-05  1.455e-04  6.489e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:35 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.562e-03  1.889e-04  6.679e-08  5.807e-02  1.404e-03 AC140005 4B D K
2023-08-07 23:35:35 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  4.231e-05  1.940e-04  7.138e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:36 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.529e-03  1.801e-04  6.404e-08  5.807e-02  1.419e-03 AC140005 4B D K
2023-08-07 23:35:36 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -6.100e-07  1.262e-04  6.370e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:37 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.496e-03  1.631e-04  6.427e-08  5.807e-02  1.434e-03 AC140005 4B D K
2023-08-07 23:35:37 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -3.556e-06  1.485e-04  6.183e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:38 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.00 -1.417e-03  1.852e-04  6.327e-08  5.807e-02  1.434e-03 AC140005 4B D K
2023-08-07 23:35:38 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.02 -8.670e-07  1.392e-04  6.137e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:39 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.02 -1.395e-03  2.204e-04  6.228e-08  5.807e-02  1.450e-03 AC140005 4B D K
2023-08-07 23:35:39 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.05 -1.408e-06  1.436e-04  6.094e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:40 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.05 -1.309e-03  2.037e-04  6.191e-08  5.807e-02  1.465e-03 AC140005 4B D K
2023-08-07 23:35:40 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.09 -3.878e-06  1.535e-04  6.078e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:41 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.09 -1.280e-03  2.365e-04  6.194e-08  5.807e-02  1.480e-03 AC140005 4B D K
2023-08-07 23:35:41 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.11 -1.337e-05  1.528e-04  6.070e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:42 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.16 -1.184e-03  1.910e-04  6.116e-08  5.807e-02  1.495e-03 AC140005 4B D K
2023-08-07 23:35:42 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.14  6.216e-06  1.440e-04  6.071e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:43 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.23 -1.142e-03  2.599e-04  6.129e-08  5.807e-02  1.495e-03 AC140005 4B D K
2023-08-07 23:35:43 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.19 -4.174e-06  1.531e-04  6.053e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:44 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.33 -1.091e-03  1.483e-04  6.063e-08  5.807e-02  1.511e-03 AC140005 4B D K
2023-08-07 23:35:44 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.25  3.163e-06  1.121e-04  6.037e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:45 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.44 -1.104e-03  3.828e-04  6.144e-08  5.807e-02  1.526e-03 AC140005 4B D K
2023-08-07 23:35:45 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.33  7.821e-06  1.833e-04  6.038e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:46 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.56 -9.928e-04  1.589e-04  6.052e-08  5.807e-02  1.541e-03 AC140005 4B D K
2023-08-07 23:35:46 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.42  1.845e-05  1.439e-04  6.031e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:47 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.70 -9.217e-04  2.078e-04  6.072e-08  5.807e-02  1.556e-03 AC140005 4B D K
2023-08-07 23:35:47 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.53  1.516e-05  1.789e-04  6.038e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
========================================================================================================================================
   Date (UTC) Time     IP Address   L St 123 567 ABCD  LP RP Score    Offset  Peer del. Peer disp.  Root del. Root disp. Refid     MTxRx
========================================================================================================================================
2023-08-07 23:35:48 fd00:1122:3344:107::e N  3 111 111 1111   0  0 0.86 -8.676e-04  2.108e-04  6.060e-08  5.807e-02  1.556e-03 AC140005 4B D K
2023-08-07 23:35:48 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.66  4.396e-06  1.446e-04  6.023e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:49 fd00:1122:3344:107::e N  3 111 111 1111   1  0 0.03 -8.400e-04  1.599e-04  6.039e-08  5.807e-02  1.572e-03 AC140005 4B D K
2023-08-07 23:35:49 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.80  1.412e-05  1.154e-04  6.009e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:50 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.95  9.523e-06  1.648e-04  6.017e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:51 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.22 -7.214e-04  1.813e-04  6.058e-08  5.807e-02  1.602e-03 AC140005 4B D K
2023-08-07 23:35:51 fd00:1122:3344:10a::e N 10 111 111 1111   1  0 0.12  1.941e-05  2.133e-04  6.027e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:53 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.42 -6.583e-04  3.170e-04  6.091e-08  5.807e-02  1.617e-03 AC140005 4B D K
2023-08-07 23:35:53 fd00:1122:3344:10a::e N 10 111 111 1111   1  1 0.31 -1.245e-06  1.976e-04  6.027e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:55 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.64 -5.054e-04  2.486e-04  6.081e-08  5.807e-02  1.648e-03 AC140005 4B D K
2023-08-07 23:35:55 fd00:1122:3344:10a::e N 10 111 111 1111   1  1 0.52  2.401e-05  2.068e-04  6.028e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:57 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.88 -3.813e-04  1.868e-04  6.060e-08  5.807e-02  1.663e-03 AC140005 4B D K
2023-08-07 23:35:57 fd00:1122:3344:10a::e N 10 111 111 1111   1  1 0.73  5.442e-06  2.082e-04  6.027e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:35:59 fd00:1122:3344:107::e N  3 111 111 1111   2  1 0.12 -2.413e-04  1.783e-04  6.044e-08  5.807e-02  1.694e-03 AC140005 4B D K
2023-08-07 23:35:59 fd00:1122:3344:10a::e N 10 111 111 1111   1  1 0.97 -1.514e-06  2.146e-04  6.028e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:01 fd00:1122:3344:10a::e N 10 111 111 1111   2  1 0.22  2.931e-05  1.467e-04  6.014e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:03 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.39 -8.028e-06  2.144e-04  6.036e-08  5.807e-02  1.740e-03 AC140005 4B D K
2023-08-07 23:36:05 fd00:1122:3344:10a::e N 10 111 111 1111   2  2 0.48  3.785e-05  2.279e-04  6.066e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:07 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.66  2.253e-04  1.704e-04  6.021e-08  5.807e-02  1.785e-03 AC140005 4B D K
2023-08-07 23:36:10 fd00:1122:3344:10a::e N 10 111 111 1111   0  2 0.00 -2.269e-03  2.181e-04  6.017e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:11 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -8.921e-04  1.838e-04  1.746e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:11 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.94  2.535e-03  2.044e-04  6.016e-08  5.807e-02  1.846e-03 AC140005 4B D K
2023-08-07 23:36:12 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -1.919e-04  1.320e-04  1.288e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:13 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  4.375e-05  1.717e-04  1.339e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:14 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  2.563e-04  1.690e-04  1.214e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:15 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  4.109e-04  2.046e-04  6.247e-07  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:15 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.97  2.963e-03  1.899e-04  6.010e-08  5.807e-02  1.892e-03 AC140005 4B D K
2023-08-07 23:36:16 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  2.689e-05  1.862e-04  7.103e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:17 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  3.333e-05  1.936e-04  6.606e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:18 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  2.997e-05  1.109e-04  6.210e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:19 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00 -1.826e-05  1.571e-04  6.435e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:19 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.08  3.035e-03  1.584e-04  6.022e-08  5.807e-02  1.205e-03 AC140005 4B D K
========================================================================================================================================
   Date (UTC) Time     IP Address   L St 123 567 ABCD  LP RP Score    Offset  Peer del. Peer disp.  Root del. Root disp. Refid     MTxRx
========================================================================================================================================
2023-08-07 23:36:20 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  3.118e-06  1.521e-04  6.313e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:21 fd00:1122:3344:10a::e N 10 111 111 1111   0  0 0.00  1.012e-06  1.925e-04  6.237e-08  0.000e+00  0.000e+00 7F7F0101 4B D K
2023-08-07 23:36:22 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00  2.666e-03  2.138e-04  6.207e-08  5.811e-02  1.373e-03 AC140005 4B D K
2023-08-07 23:36:23 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00  1.544e-03  2.201e-04  4.067e-07  5.811e-02  1.404e-03 AC140005 4B D K
2023-08-07 23:36:23 fd00:1122:3344:107::e N  3 111 111 1111   2  2 0.01  1.183e-03  1.536e-04  6.092e-08  5.807e-02  1.221e-03 AC140005 4B D K
2023-08-07 23:36:24 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00  6.364e-04  1.676e-04  2.789e-07  5.811e-02  1.434e-03 AC140005 4B D K
2023-08-07 23:36:25 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -1.264e-04  1.682e-04  3.295e-07  5.811e-02  1.465e-03 AC140005 4B D K
2023-08-07 23:36:26 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -2.879e-04  1.667e-04  2.257e-07  5.811e-02  1.495e-03 AC140005 4B D K
2023-08-07 23:36:27 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -2.443e-04  1.416e-04  3.071e-07  5.811e-02  1.526e-03 AC140005 4B D K
2023-08-07 23:36:27 fd00:1122:3344:107::e N  3 111 111 1111   1  2 0.95  5.694e-04  2.065e-04  6.530e-08  5.807e-02  1.251e-03 AC140005 4B D K
2023-08-07 23:36:28 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00  4.355e-05  1.754e-04  6.750e-08  5.811e-02  1.556e-03 AC140005 4B D K
2023-08-07 23:36:29 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -2.146e-05  1.921e-04  6.585e-08  5.811e-02  1.602e-03 AC140005 4B D K
2023-08-07 23:36:29 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.89  5.249e-04  1.953e-04  6.555e-08  5.807e-02  1.266e-03 AC140005 4B D K
2023-08-07 23:36:30 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -1.989e-05  1.700e-04  6.369e-08  5.811e-02  1.633e-03 AC140005 4B D K
2023-08-07 23:36:31 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -2.798e-05  2.132e-04  6.775e-08  5.811e-02  1.663e-03 AC140005 4B D K
2023-08-07 23:36:32 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.83  5.271e-04  1.693e-04  6.709e-08  5.807e-02  1.266e-03 AC140005 4B D K
2023-08-07 23:36:32 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00 -1.271e-05  1.687e-04  6.980e-08  5.811e-02  1.694e-03 AC140005 4B D K
2023-08-07 23:36:33 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.00  1.685e-05  1.621e-04  6.499e-08  5.811e-02  1.724e-03 AC140005 4B D K
2023-08-07 23:36:34 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.79  5.631e-04  1.442e-04  6.532e-08  5.807e-02  1.282e-03 AC140005 4B D K
2023-08-07 23:36:34 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.02 -1.641e-05  1.917e-04  6.370e-08  5.811e-02  1.755e-03 AC140005 4B D K
2023-08-07 23:36:35 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.05  4.680e-06  1.650e-04  6.256e-08  5.811e-02  1.785e-03 AC140005 4B D K
2023-08-07 23:36:36 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.73  5.884e-04  2.635e-04  6.421e-08  5.807e-02  1.297e-03 AC140005 4B D K
2023-08-07 23:36:36 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.09 -2.394e-05  1.506e-04  6.148e-08  5.811e-02  1.816e-03 AC140005 4B D K
2023-08-07 23:36:37 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.16 -1.809e-05  1.578e-04  6.125e-08  5.811e-02  1.846e-03 AC140005 4B D K
2023-08-07 23:36:38 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.70  5.994e-04  1.782e-04  6.200e-08  5.807e-02  1.312e-03 AC140005 4B D K
2023-08-07 23:36:38 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.23  1.963e-05  1.483e-04  6.095e-08  5.811e-02  1.877e-03 AC140005 4B D K
2023-08-07 23:36:39 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.33  1.415e-05  1.235e-04  6.057e-08  5.811e-02  1.907e-03 AC140005 4B D K
2023-08-07 23:36:40 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.70  6.259e-04  1.648e-04  6.114e-08  5.807e-02  1.312e-03 AC140005 4B D K
2023-08-07 23:36:40 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.44 -1.036e-05  1.317e-04  6.054e-08  5.811e-02  1.938e-03 AC140005 4B D K
2023-08-07 23:36:41 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.56  1.520e-05  1.118e-04  6.036e-08  5.811e-02  1.984e-03 AC140005 4B D K
2023-08-07 23:36:42 fd00:1122:3344:107::e N  3 111 111 1111   1  1 0.72  6.352e-04  1.401e-04  6.065e-08  5.807e-02  1.328e-03 AC140005 4B D K
2023-08-07 23:36:42 fd00:1122:3344:10a::e N  3 111 111 1111   0  0 0.70  1.451e-05  8.958e-05  6.020e-08  5.811e-02  2.014e-03 AC140005 4B D K
iliana commented 1 year ago

Maybe we ought to be looking at chrony logs on a host that has crdb running now. The error on BRM44220010 reads clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset) each time, which implies to me there was only one other node running during the time span (~23:36:09.588472 – ~23:36:28.885956) when crdb on BRM44220010 was trying to come up.

askfongjojo commented 1 year ago

The other 3 CRDB nodes all joined the cluster between 23:36:09-23:36:10

gc14 BRM42220051

root@oxz_cockroachdb_d3480152-2192-4dd9-b9c1-79d04420b039:/data/logs# egrep 'clock_monotonicity|connected to cluster' cockroach.oxzcockroachdbd3480152-2192-4dd9-b9c1-79d04420b039.root.2023-08-07T23_33_50Z.024123.log
I230807 23:33:51.450453 213 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I230807 23:33:51.457159 213 1@server/clock_monotonicity.go:140 ⋮ [n2] 19  Sleeping till wall time 1691451231457154930 to catches up to 1691451231950391215 to ensure monotonicity. Delta: 493.236285ms
I230807 23:36:09.964803 415 1@gossip/gossip.go:1547 ⋮ [n2] 644  node has connected to cluster via gossip

gc17 BRM42220017

root@oxz_cockroachdb_28f3d5aa-eb89-4ee0-ab4c-0e22d4b7aa1f:/data/logs# egrep 'clock_monotonicity|connected to cluster'  cockroach.oxzcockroachdb28f3d5aa-eb89-4ee0-ab4c-0e22d4b7aa1f.root.2023-08-07T23_36_06Z.010776.log
I230807 23:36:06.639100 217 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I230807 23:36:06.643417 217 1@server/clock_monotonicity.go:140 ⋮ [n3] 19  Sleeping till wall time 1691451366643411823 to catches up to 1691451367139041846 to ensure monotonicity. Delta: 495.630023ms
I230807 23:36:10.019871 660 1@gossip/gossip.go:1547 ⋮ [n3] 62  node has connected to cluster via gossip

gc9, BRM44220005

root@oxz_cockroachdb_dccfea99-7030-4730-b8f9-f701e9124e7c:/data/logs# egrep 'clock_monotonicity|connected to cluster' cockroach.oxzcockroachdbdccfea99-7030-4730-b8f9-f701e9124e7c.root.2023-08-07T23_33_47Z.010634.log
I230807 23:33:49.914317 122 1@server/clock_monotonicity.go:65 ⋮ [n?] 18  monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I230807 23:33:49.930192 122 1@server/clock_monotonicity.go:140 ⋮ [n1] 19  Sleeping till wall time 1691451229930189415 to catches up to 1691451230414241304 to ensure monotonicity. Delta: 484.051889ms
W230807 23:36:02.974122 1396 2@rpc/clock_offset.go:216 ⋮ [n1,rnode=2,raddr=‹[fd00:1122:3344:108::3]:32221›,class=default,heartbeat] 604  latency jump (prev avg 0.52ms, current 1.21ms)
I230807 23:36:10.818404 538 1@gossip/gossip.go:1547 ⋮ [n1] 654  node has connected to cluster via gossip

The chrony tracking log for these 3 sleds from start to the time shortly after crdb joined the cluster are as follows:

gc14 BRM42220051

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:36:01 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:36:06 fd00:1122:3344:10a::e 11      0.000 1000000.000  1.892e+03 N  1  5.300e-06  0.000e+00  9.840e-05  1.892e+03  1.500e+00
1986-12-28 00:04:34 0.0.0.0          0      0.000 1000000.000  0.000e+00 ?  0  0.000e+00 -1.892e+03  9.840e-05  2.704e-01  1.892e+03
1986-12-28 00:04:42 fd00:1122:3344:107::e 11     -9.772     12.430  3.896e+01 N  1  6.663e-06 -1.892e+03  1.119e-04  5.451e-04  1.900e+03
1986-12-28 00:04:08 fd00:1122:3344:10a::e 11     -9.772     12.430 -1.155e+09 N  1  2.981e-03 -1.931e+03  5.831e-05  3.498e+06  1.931e+03
2023-08-07 23:33:11 0.0.0.0          0     -9.772     12.430  0.000e+00 ?  0  0.000e+00  1.369e-07  5.831e-05  4.044e-03  1.155e+09
2023-08-07 23:35:22 fd00:1122:3344:10a::e 11     -9.772     12.430  4.376e-01 N  1  3.367e-02  1.379e-07  1.005e-04  3.685e-02  4.018e-01
2023-08-07 23:35:22 fd00:1122:3344:10a::e 11     -9.772     12.430  4.532e-02 N  1  2.182e-02 -4.349e-01  9.929e-05  2.615e-02  4.762e-01
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11     -9.772     12.430  1.127e-02 N  1  1.658e-02 -4.774e-01  1.099e-04  2.087e-02  5.115e-01
2023-08-07 23:35:24 fd00:1122:3344:10a::e 11     -9.772     12.430  1.020e-02 N  1  9.745e-03 -4.859e-01  9.244e-05  1.743e-02  5.159e-01
2023-08-07 23:35:25 fd00:1122:3344:10a::e 11     -9.772     12.430  4.550e-03 N  1  8.085e-03 -4.934e-01  9.786e-05  9.506e-03  5.221e-01
2023-08-07 23:35:26 fd00:1122:3344:10a::e 11     -9.710     14.742 -1.154e-02 N  1  1.247e-04 -4.952e-01  9.699e-05  1.611e-04  5.165e-01
2023-08-07 23:35:27 fd00:1122:3344:10a::e 11     -4.588     10.073 -2.703e-05 N  1  2.667e-06 -4.809e-01  1.014e-04  8.731e-06  4.839e-01
2023-08-07 23:35:28 fd00:1122:3344:10a::e 11     -3.607      7.029  3.059e-06 N  1  3.391e-06 -4.782e-01  1.106e-04  7.184e-06  4.810e-01
2023-08-07 23:35:29 fd00:1122:3344:10a::e 11     -1.618      4.303  8.292e-06 N  1  1.630e-06 -4.754e-01  7.363e-05  4.609e-06  4.782e-01
2023-08-07 23:35:31 fd00:1122:3344:10a::e 11     -1.772      2.773 -1.423e-06 N  1  1.534e-06 -4.727e-01  7.363e-05  7.021e-06  4.755e-01
2023-08-07 23:35:32 fd00:1122:3344:10a::e 11     -1.654      2.179  8.020e-07 N  1  1.560e-06 -4.699e-01  7.363e-05  5.859e-06  4.727e-01
2023-08-07 23:35:33 fd00:1122:3344:10a::e 11     -1.925      2.160 -2.085e-06 N  1  1.862e-06 -4.672e-01  7.363e-05  1.098e-05  4.700e-01
2023-08-07 23:35:34 fd00:1122:3344:10a::e 11     -1.850      1.812  9.660e-07 N  1  1.947e-06 -4.644e-01  7.363e-05  1.024e-05  4.673e-01
2023-08-07 23:35:35 fd00:1122:3344:10a::e 11     -2.260      3.280 -6.915e-06 N  1  3.852e-06 -4.617e-01  8.875e-05  6.054e-06  4.645e-01
2023-08-07 23:35:36 fd00:1122:3344:10a::e 11     -2.383      3.583  3.310e-06 N  1  3.709e-06 -4.589e-01  8.580e-05  7.734e-06  4.618e-01
2023-08-07 23:35:37 fd00:1122:3344:10a::e 11     -2.475      2.866 -4.172e-07 N  1  3.235e-06 -4.562e-01  8.525e-05  6.657e-06  4.590e-01
2023-08-07 23:35:38 fd00:1122:3344:10a::e 11     -2.397      2.303  8.712e-07 N  1  2.994e-06 -4.535e-01  8.868e-05  4.000e-06  4.563e-01
2023-08-07 23:35:39 fd00:1122:3344:10a::e 11     -2.275      1.954  1.073e-06 N  1  2.860e-06 -4.508e-01  9.137e-05  3.395e-06  4.535e-01
2023-08-07 23:35:40 fd00:1122:3344:10a::e 11     -2.285      1.517 -2.622e-07 N  1  2.482e-06 -4.480e-01  8.160e-05  2.901e-06  4.508e-01
2023-08-07 23:35:41 fd00:1122:3344:10a::e 11     -1.791      2.466 -3.739e-06 N  1  2.802e-06 -4.453e-01  6.874e-05  5.748e-06  4.481e-01
2023-08-07 23:35:42 fd00:1122:3344:10a::e 11     -1.465      2.369  3.089e-07 N  1  2.905e-06 -4.425e-01  6.874e-05  5.378e-06  4.453e-01
2023-08-07 23:35:43 fd00:1122:3344:10a::e 11     -1.420      2.003 -1.434e-07 N  1  2.987e-06 -4.398e-01  6.874e-05  9.484e-06  4.426e-01
2023-08-07 23:35:44 fd00:1122:3344:10a::e 11     -1.492      1.726 -8.904e-07 N  1  2.872e-06 -4.371e-01  6.874e-05  8.222e-06  4.399e-01
2023-08-07 23:35:45 fd00:1122:3344:10a::e 11     -1.227      1.449 -1.911e-06 N  1  2.421e-06 -4.337e-01  5.904e-05  3.368e-06  4.371e-01
2023-08-07 23:35:46 fd00:1122:3344:10a::e 11     -1.214      1.157 -2.888e-07 N  1  2.262e-06 -4.309e-01  5.904e-05  4.996e-06  4.337e-01
2023-08-07 23:35:47 fd00:1122:3344:10a::e 11     -1.195      0.957  2.365e-07 N  1  1.961e-06 -4.281e-01  6.195e-05  2.062e-06  4.310e-01
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:35:49 fd00:1122:3344:10a::e 11     -1.272      0.879 -1.181e-06 N  1  2.050e-06 -4.225e-01  6.195e-05  5.620e-06  4.282e-01
2023-08-07 23:35:51 fd00:1122:3344:10a::e 11     -1.381      0.900 -1.729e-06 N  1  2.457e-06 -4.169e-01  6.195e-05  6.779e-06  4.225e-01
2023-08-07 23:35:53 fd00:1122:3344:10a::e 11     -1.784      1.107  4.594e-06 N  1  3.003e-06 -4.114e-01  6.195e-05  8.573e-06  4.170e-01
2023-08-07 23:35:55 fd00:1122:3344:10a::e 11     -1.828      0.894  9.800e-07 N  1  3.254e-06 -4.059e-01  6.195e-05  1.159e-05  4.115e-01
2023-08-07 23:35:57 fd00:1122:3344:10a::e 11     -1.789      0.826  1.070e-06 N  1  3.463e-06 -4.004e-01  6.195e-05  1.232e-05  4.060e-01
2023-08-07 23:36:01 fd00:1122:3344:10a::e 11     -1.772      0.736  3.202e-07 N  1  3.809e-06 -3.893e-01  5.904e-05  1.684e-05  4.004e-01
2023-08-07 23:36:05 fd00:1122:3344:10a::e 11     -1.817      0.681 -1.507e-06 N  1  4.105e-06 -3.783e-01  5.904e-05  1.815e-05  3.894e-01
2023-08-07 23:36:09 fd00:1122:3344:10a::e 11     -1.816      0.921  1.179e-03 N  1  5.850e-04 -3.674e-01  1.578e-04  6.145e-04  3.784e-01
2023-08-07 23:36:10 fd00:1122:3344:10a::e 11     -1.814      1.586  8.629e-04 N  1  4.014e-04 -3.659e-01  1.071e-04  5.266e-04  3.694e-01
2023-08-07 23:36:11 fd00:1122:3344:10a::e 11     -1.804      3.049  3.288e-04 N  1  2.933e-04 -3.640e-01  1.244e-04  3.883e-04  3.675e-01
2023-08-07 23:36:12 fd00:1122:3344:10a::e 11     -1.764      6.299  2.255e-04 N  1  2.317e-04 -3.615e-01  1.169e-04  4.032e-04  3.650e-01

gc17 BRM42220017

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:03:31 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:05:38 fd00:1122:3344:10a::e 11      0.000 1000000.000 -1.155e+09 N  1  3.111e-02  0.000e+00  1.028e-04  1.158e+09  1.500e+00
2023-08-07 23:35:19 fd00:1122:3344:10a::e 11      0.000 1000000.000  4.543e-01 N  1  3.701e-05  0.000e+00  1.119e-04  8.476e-01  1.155e+09
2023-08-07 23:35:19 fd00:1122:3344:10a::e 11      0.000 1000000.000  2.672e-03 N  1  7.472e-05 -4.515e-01  1.206e-04  9.818e-01  1.892e+00
2023-08-07 23:35:20 fd00:1122:3344:10a::e 11      0.000 1000000.000  6.051e-04 N  1  5.424e-04 -4.514e-01  1.535e-04  4.448e-01  2.452e+00
2023-08-07 23:35:21 fd00:1122:3344:10a::e 11      0.000 1000000.000 -1.626e-04 N  1  5.857e-04 -4.493e-01  1.310e-04  9.256e-01  1.908e+00
2023-08-07 23:35:22 fd00:1122:3344:10a::e 11      0.000 1000000.000 -1.475e-04 N  1  5.026e-04 -4.463e-01  1.243e-04  8.348e-02  2.397e+00
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11      0.000 1000000.000  1.160e-04 N  1  5.768e-04 -4.435e-01  1.556e-04  5.671e-01  1.545e+00
2023-08-07 23:35:24 fd00:1122:3344:10a::e 11     17.219     24.962 -3.190e-04 N  1  9.987e-06 -4.408e-01  1.246e-04  3.092e-05  2.027e+00
2023-08-07 23:35:25 fd00:1122:3344:10a::e 11     19.930     19.093  1.871e-05 N  1  7.937e-06 -4.378e-01  1.037e-04  1.921e-05  4.406e-01
2023-08-07 23:35:26 fd00:1122:3344:10a::e 11     21.401     28.012 -1.231e-05 N  1  9.998e-06 -4.350e-01  1.145e-04  1.106e-05  4.379e-01
2023-08-07 23:35:27 fd00:1122:3344:10a::e 11     21.250     26.898 -4.006e-06 N  1  1.244e-05 -4.323e-01  1.503e-04  3.801e-05  4.351e-01
2023-08-07 23:35:28 fd00:1122:3344:10a::e 11     20.774     34.651 -2.227e-06 N  1  1.275e-05 -4.295e-01  1.152e-04  1.561e-05  4.324e-01
2023-08-07 23:35:29 fd00:1122:3344:10a::e 11     19.210     27.230  3.090e-06 N  1  1.295e-05 -4.268e-01  1.402e-04  3.315e-05  4.296e-01
2023-08-07 23:35:30 fd00:1122:3344:10a::e 11     17.929     16.028 -4.394e-06 N  1  1.009e-05 -4.241e-01  1.165e-04  1.276e-05  4.269e-01
2023-08-07 23:35:31 fd00:1122:3344:10a::e 11     17.263     11.816 -1.258e-06 N  1  8.445e-06 -4.213e-01  1.135e-04  1.231e-05  4.241e-01
2023-08-07 23:35:32 fd00:1122:3344:10a::e 11     17.055      9.001 -9.716e-07 N  1  8.406e-06 -4.186e-01  1.135e-04  2.134e-05  4.214e-01
2023-08-07 23:35:33 fd00:1122:3344:10a::e 11     16.935      7.205 -2.066e-06 N  1  8.052e-06 -4.158e-01  1.287e-04  1.454e-05  4.186e-01
2023-08-07 23:35:34 fd00:1122:3344:10a::e 11     16.941      5.668 -6.841e-07 N  1  7.338e-06 -4.131e-01  1.254e-04  1.321e-05  4.159e-01
2023-08-07 23:35:35 fd00:1122:3344:10a::e 11     15.701      3.945 -6.743e-06 N  1  4.802e-06 -4.103e-01  1.254e-04  1.308e-05  4.131e-01
2023-08-07 23:35:36 fd00:1122:3344:10a::e 11     17.451      4.721  1.420e-05 N  1  4.484e-06 -4.075e-01  9.986e-05  6.076e-06  4.104e-01
2023-08-07 23:35:37 fd00:1122:3344:10a::e 11     17.675      3.511 -1.677e-06 N  1  4.466e-06 -4.048e-01  9.986e-05  9.832e-06  4.076e-01
2023-08-07 23:35:38 fd00:1122:3344:10a::e 11     17.711      2.942 -6.979e-08 N  1  4.438e-06 -4.021e-01  9.986e-05  1.269e-05  4.049e-01
2023-08-07 23:35:39 fd00:1122:3344:10a::e 11     17.373      3.275 -4.382e-06 N  1  5.621e-06 -3.994e-01  9.986e-05  1.549e-05  4.022e-01
2023-08-07 23:35:40 fd00:1122:3344:10a::e 11     17.494      2.595  2.619e-06 N  1  4.610e-06 -3.966e-01  1.042e-04  7.453e-06  3.994e-01
2023-08-07 23:35:41 fd00:1122:3344:10a::e 11     17.340      2.344 -2.126e-06 N  1  4.590e-06 -3.939e-01  1.042e-04  9.641e-06  3.967e-01
2023-08-07 23:35:42 fd00:1122:3344:10a::e 11     17.150      2.261 -2.343e-06 N  1  4.871e-06 -3.912e-01  1.042e-04  9.524e-06  3.939e-01
2023-08-07 23:35:43 fd00:1122:3344:10a::e 11     17.060      1.972 -6.804e-07 N  1  4.611e-06 -3.884e-01  1.042e-04  1.137e-05  3.912e-01
2023-08-07 23:35:44 fd00:1122:3344:10a::e 11     17.004      1.770 -6.768e-07 N  1  4.535e-06 -3.857e-01  1.042e-04  1.396e-05  3.885e-01
2023-08-07 23:35:46 fd00:1122:3344:10a::e 11     17.110      1.508  1.963e-06 N  1  4.237e-06 -3.802e-01  1.076e-04  4.503e-06  3.858e-01
2023-08-07 23:35:48 fd00:1122:3344:10a::e 11     17.036      1.349 -1.730e-06 N  1  4.380e-06 -3.747e-01  1.076e-04  8.311e-06  3.803e-01
2023-08-07 23:35:50 fd00:1122:3344:10a::e 11     17.084      1.240  1.251e-06 N  1  4.620e-06 -3.692e-01  1.076e-04  1.147e-05  3.748e-01
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:35:53 fd00:1122:3344:10a::e 11     17.010      1.374 -2.145e-06 N  1  6.021e-06 -3.632e-01  1.076e-04  1.495e-05  3.693e-01
2023-08-07 23:35:55 fd00:1122:3344:10a::e 11     16.960      1.300 -8.977e-07 N  1  6.164e-06 -3.576e-01  1.076e-04  1.682e-05  3.633e-01
2023-08-07 23:35:59 fd00:1122:3344:10a::e 11     16.930      1.200 -7.386e-07 N  1  6.859e-06 -3.467e-01  1.076e-04  2.317e-05  3.577e-01
2023-08-07 23:36:03 fd00:1122:3344:10a::e 11     16.901      1.093 -9.540e-07 N  1  7.274e-06 -3.358e-01  1.076e-04  2.595e-05  3.468e-01
2023-08-07 23:36:07 fd00:1122:3344:10a::e 11     16.864      1.020 -1.397e-06 N  1  7.791e-06 -3.249e-01  1.076e-04  2.811e-05  3.359e-01
2023-08-07 23:36:15 fd00:1122:3344:10a::e 11     16.869      1.793  1.945e-03 N  1  4.302e-04 -3.029e-01  1.438e-04  5.167e-04  3.250e-01

gc9, BRM44220005

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:04:52 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:04:58 fd00:1122:3344:107::e 11     -4.105     10.228  5.519e+01 N  1  6.436e-06  0.000e+00  1.271e-04  6.412e-04  1.500e+00
1986-12-28 00:04:08 fd00:1122:3344:10a::e 11     -4.105     10.228 -1.155e+09 N  1  4.366e-03 -5.518e+01  7.560e-05  2.834e+06  5.519e+01
2023-08-07 23:33:11 0.0.0.0          0     -4.105     10.228  0.000e+00 ?  0  0.000e+00  1.526e-07  7.560e-05  6.666e-03  1.155e+09
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11     -4.105     10.228  5.015e-01 N  1  3.069e-02  1.541e-07  8.292e-05  3.311e-02  3.311e-01
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11     -4.105     10.228  3.436e-02 N  1  1.543e-02 -4.987e-01  1.182e-04  1.575e-02  5.369e-01
2023-08-07 23:35:24 fd00:1122:3344:10a::e 11     -4.105     10.228  5.992e-03 N  1  1.261e-02 -5.303e-01  8.744e-05  1.310e-02  5.548e-01
2023-08-07 23:35:25 fd00:1122:3344:10a::e 11     -4.105     10.228  6.552e-03 N  1  9.286e-03 -5.336e-01  8.923e-05  1.398e-02  5.570e-01
2023-08-07 23:35:26 fd00:1122:3344:10a::e 11     -4.105     10.228  6.387e-03 N  1  8.921e-03 -5.374e-01  6.944e-05  1.028e-02  5.625e-01
2023-08-07 23:35:27 fd00:1122:3344:10a::e 11     -4.061     12.945 -1.250e-02 N  1  2.149e-04 -5.410e-01  7.115e-05  2.181e-04  5.645e-01
2023-08-07 23:35:28 fd00:1122:3344:10a::e 11      1.397     11.489 -1.016e-04 N  1  2.357e-06 -5.258e-01  8.080e-05  1.013e-05  5.289e-01
2023-08-07 23:35:29 fd00:1122:3344:10a::e 11      3.198      2.104  3.048e-06 N  1  6.192e-07 -5.229e-01  7.115e-05  7.395e-06  5.257e-01
2023-08-07 23:35:31 fd00:1122:3344:10a::e 11      2.991      1.442 -1.198e-06 N  1  8.460e-07 -5.174e-01  7.115e-05  7.688e-06  5.230e-01
2023-08-07 23:35:32 fd00:1122:3344:10a::e 11      2.710      2.686 -6.309e-06 N  1  2.508e-06 -5.147e-01  8.219e-05  6.177e-06  5.175e-01
2023-08-07 23:35:33 fd00:1122:3344:10a::e 11      1.648      2.750 -2.865e-06 N  1  1.490e-06 -5.120e-01  6.774e-05  3.184e-06  5.148e-01
2023-08-07 23:35:34 fd00:1122:3344:10a::e 11      1.277      1.979 -1.544e-06 N  1  1.527e-06 -5.092e-01  6.774e-05  5.901e-06  5.120e-01
2023-08-07 23:35:35 fd00:1122:3344:10a::e 11      1.261      1.494  2.317e-07 N  1  1.539e-06 -5.065e-01  6.774e-05  4.635e-06  5.093e-01
2023-08-07 23:35:36 fd00:1122:3344:10a::e 11      1.265      1.027 -1.158e-08 N  1  1.188e-06 -5.037e-01  6.986e-05  2.576e-06  5.065e-01
2023-08-07 23:35:37 fd00:1122:3344:10a::e 11      1.195      1.009 -6.928e-07 N  1  1.389e-06 -5.010e-01  6.986e-05  3.656e-06  5.038e-01
2023-08-07 23:35:38 fd00:1122:3344:10a::e 11      1.220      0.938  4.628e-07 N  1  1.518e-06 -4.983e-01  6.986e-05  3.614e-06  5.010e-01
2023-08-07 23:35:39 fd00:1122:3344:10a::e 11      1.119      0.795 -1.080e-06 N  1  1.246e-06 -4.955e-01  6.396e-05  2.900e-06  4.983e-01
2023-08-07 23:35:40 fd00:1122:3344:10a::e 11      1.027      0.745 -1.006e-06 N  1  1.297e-06 -4.928e-01  6.396e-05  3.263e-06  4.956e-01
2023-08-07 23:35:41 fd00:1122:3344:10a::e 11      0.982      0.660 -3.508e-07 N  1  1.300e-06 -4.901e-01  6.396e-05  4.047e-06  4.928e-01
2023-08-07 23:35:42 fd00:1122:3344:10a::e 11      0.923      0.618 -7.403e-07 N  1  1.314e-06 -4.873e-01  6.396e-05  3.444e-06  4.901e-01
2023-08-07 23:35:43 fd00:1122:3344:10a::e 11      0.908      0.554 -4.789e-08 N  1  1.342e-06 -4.846e-01  6.396e-05  4.280e-06  4.873e-01
2023-08-07 23:35:46 fd00:1122:3344:10a::e 11      0.972      0.635  1.243e-06 N  1  1.825e-06 -4.791e-01  6.396e-05  6.208e-06  4.846e-01
2023-08-07 23:35:47 fd00:1122:3344:10a::e 11      1.061      0.676  1.345e-06 N  1  2.004e-06 -4.764e-01  6.396e-05  7.845e-06  4.791e-01
2023-08-07 23:35:49 fd00:1122:3344:10a::e 11      1.027      0.575 -1.058e-06 N  1  1.942e-06 -4.709e-01  6.396e-05  8.425e-06  4.764e-01
2023-08-07 23:35:51 fd00:1122:3344:10a::e 11      1.173      0.976  1.199e-05 N  1  4.574e-06 -4.653e-01  7.882e-05  7.094e-06  4.709e-01
2023-08-07 23:35:53 fd00:1122:3344:10a::e 11      1.302      1.086  3.068e-07 N  1  4.697e-06 -4.598e-01  7.882e-05  9.290e-06  4.654e-01
2023-08-07 23:35:55 fd00:1122:3344:10a::e 11      1.397      1.036  1.690e-06 N  1  4.898e-06 -4.543e-01  7.882e-05  1.178e-05  4.599e-01
2023-08-07 23:35:57 fd00:1122:3344:10a::e 11      1.491      1.048  2.422e-06 N  1  5.513e-06 -4.489e-01  7.882e-05  1.418e-05  4.544e-01
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:36:01 fd00:1122:3344:10a::e 11      1.504      0.929 -3.653e-08 N  1  5.874e-06 -4.380e-01  7.882e-05  1.744e-05  4.489e-01
2023-08-07 23:36:05 fd00:1122:3344:10a::e 11      1.586      0.949  3.201e-06 N  1  6.817e-06 -4.271e-01  7.882e-05  2.308e-05  4.381e-01
2023-08-07 23:36:09 fd00:1122:3344:10a::e 11      1.588      1.389  1.451e-03 N  1  5.639e-04 -4.162e-01  1.437e-04  6.338e-04  4.272e-01
2023-08-07 23:36:10 fd00:1122:3344:10a::e 11      1.593      2.332  5.173e-04 N  1  4.845e-04 -4.149e-01  1.497e-04  5.354e-04  4.185e-01
2023-08-07 23:36:11 fd00:1122:3344:10a::e 11      1.630      5.299  4.543e-04 N  1  2.790e-04 -4.127e-01  1.339e-04  3.481e-04  4.162e-01

On gc12, the sled that also has a crdb node in maintenance, ntp seemed to have caught up at 23:33:11 but went back to bigger clock skew. By the time it caught up again, the crdb smf service already exhausted its retries (all within 23:34).

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:04:23 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:04:25 fd00:1122:3344:10a::e 11    -14.387    521.398 -7.160e+00 N  1  2.688e-07  0.000e+00  8.760e-05  3.338e-03  1.500e+00
1986-12-28 00:04:33 0.0.0.0          0    -14.387    521.398  0.000e+00 ?  0  0.000e+00  7.159e+00  8.760e-05  4.689e-04  7.160e+00
1986-12-28 00:04:41 fd00:1122:3344:107::e 11    -10.766     10.727  3.895e+01 N  1  7.263e-06  7.135e+00  1.145e-04  4.861e-04  7.164e+00
1986-12-28 00:04:08 fd00:1122:3344:10a::e 11    -10.766     10.727 -1.155e+09 N  1  5.960e-03 -3.180e+01  1.159e-04  3.921e+06  3.182e+01
2023-08-07 23:33:10 fd00:1122:3344:107::e 11    -10.766     10.727 -9.466e-02 N  1  3.109e-05  1.503e-07  1.282e-04  6.392e-02  1.155e+09
2023-08-07 23:33:11 0.0.0.0          0    -10.766     10.727  0.000e+00 ?  0  0.000e+00  9.335e-02  1.282e-04  1.268e-01  2.215e-01
2023-08-07 23:35:17 fd00:1122:3344:107::e  4    -42.483     33.722 -6.770e-03 N  1  3.814e-05  6.901e-10  5.828e-02  1.264e-03  1.397e+01
2023-08-07 23:35:17 fd00:1122:3344:107::e  4    -45.810     62.415 -1.488e-03 N  1  3.701e-04  5.200e-03  5.817e-02  1.729e-03  3.720e-02
2023-08-07 23:35:18 fd00:1122:3344:107::e  4    -55.305    122.956 -3.472e-04 N  1  3.528e-04  3.970e-03  5.828e-02  1.620e-03  3.769e-02
2023-08-07 23:35:19 fd00:1122:3344:10a::e 11    -55.305    122.956  1.345e-04 N  1  2.035e-02  3.170e-03  1.223e-04  3.782e-02  3.520e-02
2023-08-07 23:35:20 fd00:1122:3344:10a::e 11    -55.305    122.956  9.612e-03 N  1  1.652e-02  2.323e-03  9.511e-05  2.157e-02  6.204e-02
2023-08-07 23:35:21 fd00:1122:3344:10a::e 11    -55.305    122.956 -1.719e-03 N  1  1.394e-02 -5.120e-03  8.782e-05  2.468e-02  5.011e-02
2023-08-07 23:35:22 fd00:1122:3344:10a::e 11    -55.305    122.956 -8.288e-03 N  1  5.032e-04 -2.847e-03  6.921e-05  5.074e-04  4.320e-02
2023-08-07 23:35:23 fd00:1122:3344:10a::e 11    -55.305    122.956 -1.202e-03 N  1  8.064e-04  2.729e-03  1.390e-04  1.071e-03  6.364e-03
2023-08-07 23:35:24 fd00:1122:3344:10a::e 11    -55.305    122.956 -8.647e-04 N  1  7.707e-04  1.212e-03  1.188e-04  1.199e-03  5.361e-03
2023-08-07 23:35:25 fd00:1122:3344:10a::e 11    -55.305    122.956  5.095e-05 N  1  5.059e-04 -1.979e-06  1.360e-04  8.796e-04  3.922e-03
2023-08-07 23:35:26 fd00:1122:3344:10a::e 11    -55.305    122.956  9.723e-04 N  1  6.037e-04 -4.923e-05  1.394e-04  6.100e-04  1.438e-03
2023-08-07 23:35:27 fd00:1122:3344:10a::e 11    -36.661    236.629  1.160e-03 N  1  3.583e-04  1.961e-06  7.462e-05  8.308e-04  1.853e-03
2023-08-07 23:35:28 fd00:1122:3344:10a::e 11     53.376    469.102  2.488e-04 N  1  2.967e-04  2.010e-06  8.514e-05  6.055e-04  2.519e-03
2023-08-07 23:35:29 fd00:1122:3344:10a::e 11    116.566    501.080 -1.944e-04 N  1  2.795e-04 -1.097e-04  9.283e-05  6.097e-04  1.538e-03
2023-08-07 23:35:30 fd00:1122:3344:10a::e 11     80.335    404.468 -2.790e-04 N  1  2.424e-04  6.778e-05  7.660e-05  5.052e-04  1.267e-03
2023-08-07 23:35:31 fd00:1122:3344:10a::e 11     58.379    524.181 -2.294e-04 N  1  2.885e-04  1.383e-05  8.139e-05  7.147e-04  1.308e-03
2023-08-07 23:35:32 fd00:1122:3344:10a::e 11    -42.630    489.885 -2.850e-04 N  1  2.189e-04  1.065e-04  9.859e-05  6.043e-04  1.547e-03
2023-08-07 23:35:33 fd00:1122:3344:10a::e 11    -43.467    336.826 -3.228e-05 N  1  2.158e-04 -9.519e-08  7.554e-05  4.521e-04  1.566e-03
2023-08-07 23:35:34 fd00:1122:3344:10a::e 11    -30.433    244.934  7.450e-05 N  1  1.725e-04  2.901e-05  6.785e-05  3.834e-04  8.632e-04
2023-08-07 23:35:35 fd00:1122:3344:10a::e 11    -23.477    193.485  3.583e-05 N  1  1.616e-04 -3.744e-05  8.354e-05  3.268e-04  7.146e-04
2023-08-07 23:35:36 fd00:1122:3344:10a::e 11     -8.988    135.692  3.130e-05 N  1  7.243e-05 -5.110e-05  8.809e-05  2.093e-04  6.409e-04
2023-08-07 23:35:37 fd00:1122:3344:10a::e 11      8.610    104.751  7.283e-05 N  1  6.015e-05 -1.982e-05  7.535e-05  1.146e-04  4.768e-04
2023-08-07 23:35:38 fd00:1122:3344:10a::e 11     14.704     76.079  2.120e-05 N  1  5.599e-05  2.402e-08  8.625e-05  7.813e-05  3.536e-04
2023-08-07 23:35:39 fd00:1122:3344:10a::e 11     13.037     50.078 -1.595e-05 N  1  4.207e-05 -1.574e-05  7.442e-05  8.981e-05  2.218e-04
2023-08-07 23:35:40 fd00:1122:3344:10a::e 11      8.218     38.780 -2.360e-05 N  1  3.740e-05  2.084e-07  6.634e-05  4.200e-05  1.790e-04
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2023-08-07 23:35:41 fd00:1122:3344:10a::e 11      7.237     33.170 -3.114e-06 N  1  4.042e-05  1.377e-05  6.634e-05  8.634e-05  1.396e-04
2023-08-07 23:35:42 fd00:1122:3344:10a::e 11     -4.995     42.953 -3.935e-05 N  1  1.353e-05  1.211e-05  1.170e-04  3.607e-05  1.714e-04
2023-08-07 23:35:43 fd00:1122:3344:10a::e 11    -11.166     19.824 -1.339e-05 N  1  9.436e-06 -1.226e-08  8.375e-05  2.152e-05  1.956e-04
2023-08-07 23:35:44 fd00:1122:3344:10a::e 11    -11.367     13.351  6.030e-07 N  1  9.666e-06  6.974e-07  8.375e-05  3.350e-05  9.834e-05
2023-08-07 23:35:45 fd00:1122:3344:10a::e 11     -9.533      9.692  1.200e-05 N  1  8.062e-06  9.381e-08  8.045e-05  9.870e-06  8.989e-05
2023-08-07 23:35:46 fd00:1122:3344:10a::e 11     -7.330      7.952  1.555e-05 N  1  7.140e-06 -2.112e-07  7.206e-05  9.082e-06  7.295e-05
2023-08-07 23:35:47 fd00:1122:3344:10a::e 11     -5.387     15.370  1.771e-05 N  1  6.114e-06  3.575e-09  7.604e-05  1.600e-05  7.031e-05
2023-08-07 23:35:48 fd00:1122:3344:10a::e 11      3.097      8.537  1.193e-05 N  1  1.843e-06  4.901e-10  7.604e-05  1.410e-05  9.454e-05
2023-08-07 23:35:49 fd00:1122:3344:10a::e 11      1.890     10.772 -8.047e-06 N  1  2.943e-06  2.246e-10  6.758e-05  1.242e-05  7.413e-05
2023-08-07 23:35:50 fd00:1122:3344:10a::e 11      1.661      8.912  6.562e-07 N  1  3.878e-06 -1.744e-09  6.758e-05  1.486e-05  6.668e-05
2023-08-07 23:35:51 fd00:1122:3344:10a::e 11      0.991      7.519 -3.501e-06 N  1  4.584e-06 -5.838e-07  9.150e-05  7.877e-06  5.931e-05
2023-08-07 23:35:52 fd00:1122:3344:10a::e 11      0.878      5.850 -2.052e-07 N  1  4.921e-06  1.663e-06  6.758e-05  2.720e-05  6.537e-05
2023-08-07 23:35:53 fd00:1122:3344:10a::e 11      0.821      4.635 -3.175e-07 N  1  4.940e-06  1.397e-06  6.758e-05  2.818e-05  6.984e-05
2023-08-07 23:35:54 fd00:1122:3344:10a::e 11      0.397      4.092 -2.629e-06 N  1  4.712e-06  1.316e-06  6.758e-05  2.510e-05  6.944e-05
2023-08-07 23:35:55 fd00:1122:3344:10a::e 11     -1.038      5.534 -1.318e-05 N  1  6.311e-06  1.781e-06  1.144e-04  7.364e-06  6.803e-05
2023-08-07 23:35:56 fd00:1122:3344:10a::e 11     -1.714      4.783 -1.139e-06 N  1  6.383e-06 -3.464e-09  1.144e-04  1.476e-05  8.684e-05
2023-08-07 23:35:57 fd00:1122:3344:10a::e 11     -2.214      3.934 -4.027e-06 N  1  6.017e-06  1.001e-06  1.236e-04  8.160e-06  7.909e-05
2023-08-07 23:35:58 fd00:1122:3344:10a::e 11     -2.267      3.115  1.650e-07 N  1  5.664e-06  2.263e-06  6.758e-05  3.646e-05  8.003e-05
2023-08-07 23:35:59 fd00:1122:3344:10a::e 11     -2.402      2.685 -1.588e-06 N  1  5.421e-06  1.577e-06  6.758e-05  3.516e-05  7.650e-05
2023-08-07 23:36:00 fd00:1122:3344:10a::e 11     -2.413      2.206  1.439e-07 N  1  4.972e-06  1.938e-06  6.758e-05  2.920e-05  7.584e-05
2023-08-07 23:36:01 fd00:1122:3344:10a::e 11     -2.591      1.980 -2.570e-06 N  1  4.682e-06  1.361e-06  6.758e-05  2.865e-05  6.801e-05
2023-08-07 23:36:02 fd00:1122:3344:10a::e 11     -2.519      1.754  1.598e-06 N  1  4.623e-06  1.740e-06  6.758e-05  2.980e-05  6.940e-05
2023-08-07 23:36:03 fd00:1122:3344:10a::e 11     -2.624      1.531 -1.926e-06 N  1  4.220e-06  1.392e-07  1.071e-04  6.175e-06  6.651e-05
2023-08-07 23:36:04 fd00:1122:3344:10a::e 11     -2.646      1.349 -8.227e-08 N  1  4.124e-06  1.387e-06  6.758e-05  2.590e-05  6.436e-05
2023-08-07 23:36:06 fd00:1122:3344:10a::e 11     -2.620      1.239  5.814e-07 N  1  4.362e-06  7.759e-07  6.758e-05  2.662e-05  6.587e-05
2023-08-07 23:36:08 fd00:1122:3344:10a::e 11     -2.618      1.976  1.254e-03 N  1  3.742e-04  1.887e-07  1.327e-04  4.310e-04  6.510e-05
2023-08-07 23:36:09 fd00:1122:3344:10a::e 11     -2.603      4.481  1.153e-03 N  1  2.630e-04  2.334e-06  1.408e-04  3.560e-04  1.980e-03
2023-08-07 23:36:10 fd00:1122:3344:10a::e 11     -2.518     10.540  2.306e-04 N  1  2.300e-04  2.064e-06  1.195e-04  4.994e-04  1.992e-03
2023-08-07 23:36:11 fd00:1122:3344:10a::e 11     -2.339     19.119  6.301e-05 N  1  2.617e-04 -7.694e-05  1.235e-04  3.735e-04  1.217e-03

I still don't know what can be concluded from the data above as it relates to the new update procedures for the rack. It's possible that we ran into this issue in the past but just didn't know it (not very likely because I often query crdb and never hit this particular issue), or it's something more likely to happen when we do not delete the switch zones and there is external connectivity for the two boundary NTP zones as soon as they come up.

mlichvar commented 1 year ago

The tracking logs show large "remaining correction" in hundreds of milliseconds. That's the correction made slowly by slewing. A faster correction by stepping in later updates can be enabled by increasing the second number in the makestep directive, or setting it to -1 to allow step on any update.

iliana commented 1 year ago

It looks like CRDB started running on gc14/gc17/gc9 while the remaining correction (which I think is equivalent to the System time field in chronyc tracking) was -0.05 < corr < 0.05, which would make this condition pass:

https://github.com/oxidecomputer/omicron/blob/3df6222f1780fead403ca27b383db256ffea9984/sled-agent/src/services.rs#L2352-L2365

But at the time CRDB was running, the current reference was 0.0.0.0. We're not correctly checking to see if chrony is synchronized to any particular server; this is I think what skew != 0.0 is checking for, but in the logs above we can see that the skew is non-zero when the reference is 0.0.0.0.

jclulow commented 1 year ago

FWIW, this is what I threw together for diplomat, which wants to act on chrony status: https://github.com/oxidecomputer/helios-diplomat/blob/6a22fc85fb1e587ae55a56ea54ba07154bb7208a/diplomat/src/main.rs#L47