search

projectbuendia / buendia

Main project repository (see the Wiki for details)
Apache License 2.0
117 stars 36 forks source link

Server clock gets reset to 2016-01-01 sometimes #226

Closed zestyping closed 4 years ago

zestyping commented 5 years ago

The result is catastrophic, causing clients to sync forever, running up load on the server, causing it to run out of memory, and preventing clients from ever reaching a stable state.

The message log shows that during boot, the CMOS clock is read as 2016-01-01, and then the clock is adjusted to 2016-11-03 at a later point during startup.

zestyping commented 5 years ago

This does not happen on every boot; it appears to be random.

zestyping commented 5 years ago

Here's the output of grep 'setting system clock' messages. Each line corresponds to one reboot. The lines are sequential even though the timestamps are not (because they come from the sometimes incorrect system clock). You can see that the clock sometimes tracks the correct date and time and sometimes gets reset to 2016-01-01 or 2016-11-03.

Nov  4 17:05:16 buendia kernel: [    1.061219] rtc_cmos 00:03: setting system clock to 2016-11-04 17:05:13 UTC (1478279113)
Oct  2 16:21:20 buendia kernel: [    1.051241] rtc_cmos 00:03: setting system clock to 2019-10-02 16:21:17 UTC (1570033277)
Nov  3 17:16:42 buendia kernel: [    1.057654] rtc_cmos 00:03: setting system clock to 2016-01-01 00:00:15 UTC (1451606415)
Nov  3 17:37:54 buendia kernel: [    1.049988] rtc_cmos 00:03: setting system clock to 2016-11-03 17:37:51 UTC (1478194671)
Oct  3 17:00:21 buendia kernel: [    1.048384] rtc_cmos 00:03: setting system clock to 2019-10-03 17:00:18 UTC (1570122018)
Oct  3 20:13:26 buendia kernel: [    1.050928] rtc_cmos 00:03: setting system clock to 2019-10-03 20:13:24 UTC (1570133604)
Oct  3 22:43:55 buendia kernel: [    1.058551] rtc_cmos 00:03: setting system clock to 2019-10-03 22:43:52 UTC (1570142632)
Oct  4 07:47:08 buendia kernel: [    1.058335] rtc_cmos 00:03: setting system clock to 2019-10-04 07:47:05 UTC (1570175225)
Nov  3 17:16:42 buendia kernel: [    1.049575] rtc_cmos 00:03: setting system clock to 2016-01-01 00:00:15 UTC (1451606415)
Nov  3 17:36:23 buendia kernel: [    1.051798] rtc_cmos 00:03: setting system clock to 2016-11-03 17:36:21 UTC (1478194581)
Nov  3 17:16:42 buendia kernel: [    1.060639] rtc_cmos 00:03: setting system clock to 2016-01-01 00:00:15 UTC (1451606415)
Nov  3 17:16:42 buendia kernel: [    1.048728] rtc_cmos 00:03: setting system clock to 2016-01-01 00:01:22 UTC (1451606482)
Nov  3 20:27:23 buendia kernel: [    1.060827] rtc_cmos 00:03: setting system clock to 2016-11-03 20:27:21 UTC (1478204841)
Oct  4 14:42:23 buendia kernel: [    1.051297] rtc_cmos 00:03: setting system clock to 2019-10-04 14:42:21 UTC (1570200141)
Oct  4 14:43:49 buendia kernel: [    1.062398] rtc_cmos 00:03: setting system clock to 2019-10-04 14:43:47 UTC (1570200227)
Oct  5 06:51:01 buendia kernel: [    1.049531] rtc_cmos 00:03: setting system clock to 2019-10-05 06:50:59 UTC (1570258259)
Oct  5 07:01:38 buendia kernel: [    1.051131] rtc_cmos 00:03: setting system clock to 2019-10-05 07:01:36 UTC (1570258896)
Oct  5 07:17:51 buendia kernel: [    1.050474] rtc_cmos 00:03: setting system clock to 2019-10-05 07:17:49 UTC (1570259869)
Oct  5 07:41:34 buendia kernel: [    1.050119] rtc_cmos 00:03: setting system clock to 2019-10-05 07:41:32 UTC (1570261292)
Oct  5 08:03:49 buendia kernel: [    1.061516] rtc_cmos 00:03: setting system clock to 2019-10-05 08:03:47 UTC (1570262627)
Oct  5 08:22:54 buendia kernel: [    1.061217] rtc_cmos 00:03: setting system clock to 2019-10-05 08:22:52 UTC (1570263772)
Oct  5 08:41:33 buendia kernel: [    1.057288] rtc_cmos 00:03: setting system clock to 2019-10-05 08:41:31 UTC (1570264891)
Oct  5 08:49:08 buendia kernel: [    1.056907] rtc_cmos 00:03: setting system clock to 2019-10-05 08:49:06 UTC (1570265346)
Oct  5 08:55:22 buendia kernel: [    1.048690] rtc_cmos 00:03: setting system clock to 2019-10-05 08:55:20 UTC (1570265720)
Oct  5 09:08:41 buendia kernel: [    1.048648] rtc_cmos 00:03: setting system clock to 2019-10-05 09:08:39 UTC (1570266519)
Oct  5 09:27:48 buendia kernel: [    1.058037] rtc_cmos 00:03: setting system clock to 2019-10-05 09:27:46 UTC (1570267666)
Oct  5 09:35:02 buendia kernel: [    1.048610] rtc_cmos 00:03: setting system clock to 2019-10-05 09:35:00 UTC (1570268100)
Oct  5 09:37:36 buendia kernel: [    1.062848] rtc_cmos 00:03: setting system clock to 2019-10-05 09:37:34 UTC (1570268254)
Oct  5 09:43:44 buendia kernel: [    1.059960] rtc_cmos 00:03: setting system clock to 2019-10-05 09:43:42 UTC (1570268622)
Oct  5 10:44:51 buendia kernel: [    1.060210] rtc_cmos 00:03: setting system clock to 2019-10-05 10:44:49 UTC (1570272289)
Oct  5 11:11:46 buendia kernel: [    1.058887] rtc_cmos 00:03: setting system clock to 2019-10-05 11:11:44 UTC (1570273904)
Oct  5 11:27:58 buendia kernel: [    1.063364] rtc_cmos 00:03: setting system clock to 2019-10-05 11:27:56 UTC (1570274876)
Oct  5 11:57:37 buendia kernel: [    1.050545] rtc_cmos 00:03: setting system clock to 2019-10-05 11:57:35 UTC (1570276655)
Nov  3 17:16:42 buendia kernel: [    1.064488] rtc_cmos 00:03: setting system clock to 2016-01-01 00:00:17 UTC (1451606417)
Oct  7 15:44:13 buendia kernel: [    1.061281] rtc_cmos 00:03: setting system clock to 2019-10-07 15:44:11 UTC (1570463051)
Oct 18 19:49:38 buendia kernel: [    1.057549] rtc_cmos 00:03: setting system clock to 2019-10-18 19:49:36 UTC (1571428176)
zestyping commented 5 years ago

And here are the messages from two boot sequences, one with a properly set clock and one with an incorrect clock. I don't see any substantive differences between them.

goodclock.txt badclock.txt