search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.37k stars 602 forks source link

Clockdrift in the created guest #850

Open dirkdev98 opened 2 years ago

dirkdev98 commented 2 years ago

Description

Running Lima via Colima (https://github.com/abiosoft/colima/), I have occasional clock drift issues. I am unable to find a cause for this or even a way to reproduce it. It seems to be after I lock my laptop and get back the next morning, but only once in a while (every 10 days or so) and not consistently every day.

Versions:

❯ uname -a
Darwin Dirks-MacBook-Pro.local 21.4.0 Darwin Kernel Version 21.4.0: Mon Feb 21 20:36:53 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T8101 arm64

❯ lima --version
limactl version 0.9.2

❯ colima version  
colima version 0.3.4
git commit: 5a4a70481ca8d1e794677f22524e3c1b79a9b4ae

runtime: docker
arch: aarch64
client: v20.10.14
server: v20.10.11

docker run -it --rm --cap-add SYS_TIME --privileged -e ALLOW_CIDR=0.0.0.0/0 -p 123:123/udp geoffh1977/chrony

Writing New Config File

2022-05-07T08:00:43Z chronyd version 4.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP -SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
2022-05-07T08:00:43Z Could not read valid frequency and skew from driftfile /var/lib/chrony/chrony.drift
2022-05-07T08:00:43Z Initial frequency -100000.000 ppm
2022-05-07T08:00:47Z System's initial offset : 978.140307 seconds slow of true (step)
2022-05-07T08:00:52Z Backward time jump detected!
2022-05-07T08:02:00Z Selected source 162.159.200.1 (pool.ntp.org)
2022-05-07T08:02:00Z System clock wrong by 978.685231 seconds
2022-05-07T08:18:18Z System clock was stepped by 978.685231 seconds
2022-05-07T08:02:57Z Backward time jump detected!
2022-05-07T08:02:57Z Can't synchronise: no selectable sources
2022-05-07T08:05:07Z Selected source 94.198.159.15 (pool.ntp.org)
2022-05-07T08:05:07Z System clock wrong by 977.327132 seconds
2022-05-07T08:21:24Z System clock was stepped by 977.327132 seconds
2022-05-07T08:05:06Z Backward time jump detected!
2022-05-07T08:05:06Z Can't synchronise: no selectable sources

References https://github.com/abiosoft/colima/issues/274, https://github.com/lima-vm/lima/pull/490

afbjorklund commented 2 years ago

Possibly alpine-lima is missing an NTP server ? I thought that lima was adding a RTC in qemu ?

For Ubuntu (systemd) I think it is being handled by systemd-timesyncd, checking with /dev/rtc0

jandubois commented 2 years ago

Possibly alpine-lima is missing an NTP server ? I thought that lima was adding a RTC in qemu ?

It does indeed (in #490). For some reason I remembered that it would sync time with the host instead. So it looks like the RTC gets out of sync as well.

jandubois commented 2 years ago

Here is how Docker Desktop handles this: Addressing Time Drift in Docker Desktop for Mac.

vielmetti commented 2 years ago

I'm seeing something similar in a different configuration (details at #939).

AndreasA commented 1 year ago

The issue still seems to exist. Any timeframe regarding a solution because recently I ran into that issue upon validating a JWT token (not before / expires etc.). the time difference was actually only about 0.4 seconds but that can be enough for its timestamps to not match anymore and validation to fail. It also seems to exist upon using macOS HyperKit (new virtualization framework).

using busybox as ntp (as mentioned in the article referenced in the comment https://github.com/lima-vm/lima/issues/850#issuecomment-1121980785) seems to work best. there still is a bit of drift but it seems to be the lowest one for now.

outcoldman commented 1 year ago

Created 2 VMs yesterday, woke up to way more drammatic shift in time 

Wed Oct 19 04:16:54 UTC 2118

and

Wed Dec 15 06:03:40 UTC 2213
outcoldman commented 1 year ago

The worst thing, for some reason I cannot even set the date manually

hwclock --hctosys
hwclock: settimeofday() failed: Invalid argument
date -s '2023-09-01 01:00:00'
date: cannot set date: Invalid argument
Fri Sep  1 01:00:00 UTC 2023
outcoldman commented 1 year ago

To fix this issue. I have rebooted both VMs, that sync the dates, and I did after

timedatectl set-ntp no
apt install ntp

Hope that is going to fix this issue for now.

I am on lima 0.17.2, and using ubuntu 23.04/release-20230810/ubuntu-23.04-server-cloudimg-arm64.img

Running with vz on macOS Sonoma.

krlmlr commented 1 year ago

Working around (on Colima):

colima ssh -- sudo date -s @$(TZ=UTC date "+%s")

It's only accurate to the second, that's enough for my use case.