Time synchronization

[bgilbert]

Feature Request

Environment

Bare metal

Desired Feature

Ignition runs early in boot, before any time synchronization, and runs on newly-installed systems which may not have an accurate system clock. This can cause TLS certificate validation failures during fetch.

Consider adding a mechanism to query time servers and use the result for TLS certificate validation. This might be SNTP, roughtime, or HTTP Date headers.

Firewalls might not allow access to public time servers, so we can't just hardcode a server and forget about it. We can allow configuring a time server in an Ignition config, but if the initial config is fetched over HTTPS, there's a bootstrapping issue. So this might not be sufficiently practical.

Other Information

crypto/tls.Config includes a callback for getting the current time, so we can implement this without changing the system clock.

[ajeddeloh]

Several options here: 1) Add something to the spec to tell ignition what ntp servers to use 2) Add something to the initramfs to set the system time before Ignition runs 3) Add something to the initramfs to tell Ignition what time it started without setting system time 4) Add something to the spec, but allow setting a default via a karg

Option 1 is nice since it is flexible and doesn't change the system time, but would require a spec change. It also has a chicken-and-egg problem of what to do if the initial config is fetched over https. It does mean Ignition would need to learn about NTP which isn't ideal.

Option 2 is less flexible, though could be made adequately flexible through kargs. It avoids the chicken-and-egg problem, but sets the system time which may not be what the user wants

Option 3 is like option 2, but we query the time right before running Ignition, then tell it the time it started and it can calculate time from there. I prefer option 3 over option 2.

Option 4 is relatively clean and I think I prefer it to option 1. It avoids the chicken-and-egg problem but still means Ignition needs to learn about ntp.

[cgwalters]

xref https://github.com/openshift/machine-config-operator/issues/629

[bgilbert]

I like option 4. I don't think HTTP is less secure than NTP, so arguably we could just use the HTTP Date header.

For bootstrapping I think there are two cases:

1. Bare metal systems (except live ISO) receive an ignition.config.url kernel argument anyway, so the user can pass a second karg.
2. On live ISO and cloud platforms other than Packet, if this functionality is even needed, fetching the initial Ignition config doesn't require HTTPS so a config option would work.

[ajeddeloh]

wrt openshift and spec changing: all changes need to go in the 2.4.0-experimental or 3.1.0-experimental spec, so openshift would need to use those.

Does packet set the clocks on their servers when you get a machine? Might just not be an issue there.

[cgwalters]

For bare metal (or virt acting as metal) systems, one thing we could do is take the flow of:

• firstboot from e.g. ISO w/embedded ignition config (no remote resources with TLS+timestamp checks)
• Our userspace chrony runs and syncs to correct time
• coreos-installer learnds to capture the time and inject it into e.g. /boot/coreos.timestamp (or...hmm, you know, we could just set the time to be no earlier than that of the creation timestamp on /boot/config.ign automatically by default)
• Add code to our initramfs that syncs up to at least that time

[cgwalters]

Longer term though I think for CoreOS at least we need to move our time sync logic into the initramfs by default. On some platforms, there's a correct thing to do by default (major public clouds use a link-local NTP server): https://github.com/coreos/fedora-coreos-tracker/blob/master/internals/README-internals.md#time-synchronization

Ideally hypervisors like VSphere and Hyper-V provide a similar channel.

[bgilbert]

In particular, this is relevant for the Raspberry Pi, which doesn't have a battery-backed RTC.

[djds]

This also has the impact of failing to set up disk encryption on a Raspberry Pi CM4 board if any tang server is behind TLS even when paired with a separate tang server running over unencrypted HTTP with a threshold of 1. When paired with the official I/O board and correct overlay, it's possible to use the board RTC. Unfortunately ignition runs before that's synced back.

[ibotty]

This makes raspberries pretty unsuitable to be installed with coreos (if anything remote is to be done). Is there a workaround?

[jlebon]

One hack for the Pi if you have remote HTTPS resources is to use systemd.clock_usec= when flashing the card with coreos-installer install (it takes a --append-karg option). That assumes of course that you'll boot the system not long after flashing.

You'll want to add a systemd unit that deletes that karg on first boot. (I guess for the TLS Tang case, you could have a systemd service that bumps the karg on reboot, but I wouldn't rely on this setup -- at least for the first boot case, you can monitor the system to tell whether the hack worked.)