IaaS DR/Standard on NTP service

[garloff]

As a user, I want my VMs to always have correct system time.

Definition of Ready:

• [x] User Story is small enough to be finished within one sprint
• [x] User Story is clear and understood by the whole team
• [x] Acceptance criteria are defined
• [x] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [ ] All acceptance criteria are met
• [ ] Changes have been reviewed
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

[garloff]

Correct system time is important for TLS cert validation and log correlation and is considered good hygiene for any system. Many VM images come with preconfigured NTP service (from public NTP sources such as pool.ntp.org). However, this is an incomplete solution with a few disadvantages:

• It does not work for VMs without outbound internet access
• It makes precision and reliability a bit lower than it could be (by having a reliable local source)
• It does expose a tiny bit of information to the NTP server operator
• It controls the NTP server in the images, which should be cloud-independent -- yet the best NTP source is dependent on the cloud's geographic location (or maybe more precisely it's position in the internet topology)

Better would be a reliable local clock / NTP service that the operator provides.

[garloff]

One option would be to mandate that the host (hypervisor) clock is always reliable (via NTP) and the guests get the time via kvm-clock.

This option unfortunately sounds simpler than it is in practice -- many pitfalls exist that can prevent this from working as reliably as it should.

[garloff]

Another option is that the cloud provider has an NTP server available in every subnet (like it should do with a DNS server, see #229). The NTP server then can be fed to the VMs via a DHCP option. The NTP server could also be injected into the VM using the vendor_data mechanism (which can be opted out when a VM is created).

To my knowledge, providing NTP service into the local subnet is not implemented by default in existing OpenStack clouds, so this may need implementation work. (OTC provides an NTP source, but that is in a different subnet and thus requires a router to be connected. The IP is preconfigured in the provider-specific images which only some customers use.)

The mechanism via DHCP and/or via vendor-data needs to be prototyped and tested -- dhcp clients may be configured to not consumed the NTP options by default on some operating systems, e.g. If it does work relatively reliably, the dhcp option to feed the information would be preferrable (as it is using a standard as opposed to do scripting that is highly dependent on the OS image configuration to work).

[kgube]

Statically adding a local NTP server to all dhcp-replies looks pretty straight-forward using the dnsmasq.conf (or the ovn_dhcp4_global_options, when using ovn built-in dhcp). I'm still looking into how this would work with the different IPv6 address configuration and router advertisement modes.

Of course this requires the static NTP server to be reachable by all VMs, which might not be the case for subnets without gateway. As Kurt mentioned above, injecting an NTP server into subnets seems to not be supported currently. What neutron does support though is injecting dhcp agent ports (with subnet-specific IPs) and the metadata service (with the static IP 169.254.169.254/fe80::a9fe:a9fe, routed either via the gateway, or one of the dhcp IPs if no gateway is present). I'm currently trying to figure out if it would be feasible to provide NTP service via the metadata service IP, so we could maybe just reuse the existing injection-magic and also have a static IP that could be added to all the dhcp replies.

[kgube]

It seems that configuring NTP servers through vendor data may actually be better supported than using DHCP. At least cloud-init has a module for NTP: https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ntp

We did have a discussion today, to which degree cloud-init can be considered a standard for cloud images, and whether it might make sense to require it for SCS. The conclusion was that, rather than requiring cloud-init, the CSPs should make sure that whatever method of time synchronization they provide, is supported by all the images they offer.

This is also reflected in the current draft for the DR.

[kgube]

I looked some more into dhcp, openstack metadata, cloud-init and possible alternatives.

OpenStack can provide instances with metadata (interface-configuration, hostname, ssh-keys, ...), user-data, and vendor-data. Metadata is standardized and provided both in an EC2-compatible and an OpenStack-specific format, though neither of those include NTP-configuration. User-data is a single unspecified blob of data, vendor-data has a JSON format but is not specified further, their specific interpretation is up to the booted instance. When handled by cloud-init, they can contain scripts to be executed directly (and could be used to configure NTP), or cloud-init's own yaml-based cloud-config format, which supports an NTP configuration clause and can apply that to a number of NTP-clients.

The majority of Linux cloud images do include cloud-init. Of the distributions supported by the openstack diskimage-builder, only Gentoo does not include it by default [1].

Exceptions are usually images optimized for size or boot-time:

• Alpine provides cloudinit and tiny images, the latter of which replace cloud-init with their own tiny-cloud project [2], which does not seem to support vendor data, including NTP config.
• The Fedora/RHEL CoreOS images use Afterburn [3] to handle metadata for basic network setup and SSH key injection and Ignition [4] to handle user-data, configure partitioning and inject arbitrary files. They do not seem to handle cloud-config data. Ignition could probably be configured to inject NTP config from a provider-controlled source, but in the default images it isn't.
• micro-config-drive (ucd) is part of the Clear Linux project [5] and supports openstack meta- and user-data, including a subset of the cloud-config format, though not as vendor-data and without NTP support.
• CirrOS includes its own init scripts that support ec2 and configdrive sources for meta-data and script-based user-data [8], but have no support for vendor-data. (CirrOS images also don't even include an NTP client.)

There is no Windows support for cloud-init, though there is cloudbase-init [6], which aims to provide some compatibility with openstack metadata and user-data, though not vendor data. It also has no support for setting NTP servers, only for enabling or disabling the use of DHCP-provided NTP-servers.

After looking into cloud-init and it's competitors more, using DHCP to provide NTP-servers does look a bit more attractive. A lot of cloud images do seem to support DHCP-provided NTP servers, though I would like to test this more systematically. There is also still the problem of DHCPv6, which at least in the OVN-implementation does not seem to support the NTP-server option [7].

[1] https://docs.openstack.org/diskimage-builder/latest/elements/cloud-init/README.html [2] https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud [3] https://coreos.github.io/afterburn/ [4] https://coreos.github.io/ignition/ [5] https://github.com/clearlinux/micro-config-drive [6] https://cloudbase.it/cloudbase-init/ [7] https://docs.openstack.org/neutron/latest/ovn/dhcp_opts.html [8] https://github.com/cirros-dev/cirros/tree/main/src/sbin

[kgube]

The NTP option on OVN's DHCPv6 implementation seems to be missing not just in the neutron drivers, but also in the OVN code itself: https://github.com/ovn-org/ovn/blob/main/lib/ovn-l7.h#L265

[kgube]

I finally managed to build a working test environment for the dhcp options. (If anyone wants to replicate this, DHCPv4's NTP-server option only allows IPv4 addresses as value (DHCPv6 does allow both addresses and FQDNs), but openstack does not validate this, leading to undefined behavior. Dnsmasq will ignore the option if the value is incorrect, but OVN will stop serving dhcp for affected ports.)

So I tried a number of images to see if they would use NTP servers provided via DHCPv4:	Image	NTP Client	NTP Server from DHCP
Ubuntu 22.04 LTS	systemd-timesyncd	yes
Debian 12 (bookworm)	systemd-timesyncd	yes
Fedora 38	chrony	yes¹
AlmaLinux 8	chrony	yes¹
Fedora CoreOS 40	chrony	yes¹
Alpine 3.19.1 tiny (for AWS)	chrony	no²
CirrOS 0.6.2	-	no

¹ chrony supports combining multiple time sources to increase precision and reliability. The DHCP-provided NTP server will typically be added to a list of time servers included in the image. ² The image explicitly disables this and hard-codes 169.254.169.123 as NTP server. This seems to be AWS-specific, Alpine images for other cloud providers may handle that differently.

[kgube]

I looked into how different hyperscalers handle time synchronization:

• As hinted in my previous comment, AWS provides a link-local NTP-server that is reachable under 169.254.169.123 or fd00:ec2::123 and appears to frequently be hard-coded into AWS-specific images. It's unclear whether the NTP-servers are also provided via DHCP. For some combinations of flavors and regions there is also support for a para-virtualized PTP hardware clock, provided by the ENA kernel driver [1].
• GCP provides NTP service through metadata.google.internal which resolves to 169.254.169.254 [2]. It is implied in the ntpd example that the server is also provided via DHCP.
• Azure has no internal NTP server, but also provides a para-virtualized PTP clock through the hv_utils kernel module in their Linux integration services (LIS) [3].

I also did some research to get a better understanding of the different components in the linux time subsystem [4] and the differences between clocksources, real-time clocks and system clock, and also to get some background on PTP hardware clocks in linux [5], which seem to be a common target for RTC para-virtualization (such as KVMs ptp_kvm driver).

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#connect-to-the-ptp-hardware-clock [2] https://cloud.google.com/compute/docs/instances/configure-ntp [3] https://learn.microsoft.com/en-us/azure/virtual-machines/linux/time-sync [4] https://www.kernel.org/doc/Documentation/timers/timekeeping.txt [5] https://www.kernel.org/doc/html/latest/driver-api/ptp.html

[kgube]

Some more background on the Linux system time:

At boot, the system time is synchronized with a real time clock (RTC) [1], if one is available. The RTC provides an absolute time, including the current year and date. In hardware, the RTC is typically a battery-backed quartz-clock on the mainboard and has a limited resolution (I read something about 10-20 ms). Qemu can use different sources to emulate an RTC, but typically the system time of the host is used. Other Hypervisors may handle this differently though, and it is unclear if all of them even provide an emulated RTC.

After the initialization, a high-resolution monotonous counter clocksource is used to advance the system time [2]. Typically, the timestamp counter (TSC) of the CPU is used for this purpose, which offers a resolution in the nanosecond range. Because it is much less stable, Linux maintains an offset that can be changed to re-sync the the system time to an RTC or other sources. KVM provides the para-virtualized kvm-clock clocksource that, among other features, supports clock synchronization on migration between different hosts [3].

[1] https://www.kernel.org/doc/Documentation/rtc.txt [2] https://www.kernel.org/doc/Documentation/timers/timekeeping.txt [3] https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt

[kgube]

What are our requirements from an SCS perspective?

1. We want users to know what to expect from an SCS cloud, and to be able to migrate between their applications between SCS clouds with minimal overhead. That means we should decide on a common approach for time synchronization, ideally one that would allow users to re-use cloud images between SCS clouds.

2. We want to enable a wide range of cloud providers to adopt SCS with minimal overhead. So, we should prefer solutions that are not dependend on specific backends of openstack service, or even specific implementations of the openstack API. In the context of time synchronisation that means we should avoid relying on hypervisor-specific features, such as para-virtualized clocks. This does not mean that CSPs shouldn't provide such features if they are able to, just that users should not rely on their presence.

In this context it probably makes sense to distinguish between default images provided by CSP and images provided by users. Provider-images should really use whatever works best in their specific cloud, including para-virtualized time-sync or hardcoded local NTP servers. User-images should target a portable method of time-sync, such as a local NTP server provided via DHCP.