OS X, podman machine time stops sometime

[dm3ch]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description Date output is wrong for both containers and podman machine itself on OS X.

Steps to reproduce the issue: I'm not sure 100% in my reproduction guide

1. Install podman on OSX
2. Create podman machine
3. Wait couple of days
4. Get date from podman machine

Describe the results you received: OS X date:

$ date
Sun Sep 12 15:55:30 MSK 202

Podman container and podman machine ssh date:

$ date
Fri Sep 10 19:47:08 UTC 2021

I have ran same command again and time was completely the same. After stoping and starting machine again time started to go.

Describe the results you expected: Date inside podman machine should be right

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md) Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 22:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 23:46:36 2021
OS/Arch:      linux/amd64

OS X info:

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[mayurah]

@ashley-cui / @rhatdan Sadly this is still an issue after over a year.

While this gets fixed, should we mark podman as beta for macOS users?

Took me on an unbelievable ride on three instances to triage what's wrong with signature based auth and log event timing.

• Developing Slack Bot which requires accurate time
• Sending Splunk log events resulted in strange time delays
• OTP Auth generation resulted in invalid code

Will try several options posted here and see if anything helps with internal QEMU VM. Do share any workaround if you have any!

[ashley-cui]

@mayurah A fix for this is on our roadmap. In the meantime, using podman machine ssh to sync time is the best workaround.

https://github.com/containers/podman/issues/11541#issuecomment-1188987085

[mayurah]

@mayurah A fix for this is on our roadmap. In the meantime, using podman machine ssh to sync time is the best workaround.

#11541 (comment)

The issue is that you have to apply band-aid fix each time macOS M1 wakes up from sleep! If underlaying qemu VM had auto-NTP (which isn't hard to setup inside VM), it would probably solve the issue.

For a perm fix, would you be able to take a PR if raised and review it? or is it that someone is actually working on the fix?

I believe the fix would help a lot of users who doesn't even know that this is a big issue for applications that relies on time!

Alternatives such as Docker Desktop has become expensive possession to have for infrequent container building efforts.

Please guide!

[gbraad]

@cfergeau can you share some information related to the vfkit implementation to resolve this? This can be adopted for use with qemu quite 'easily'.

[cfergeau]

It's using a combination of sleepwatcher running on the host to detect the mac waking up from sleep. This triggers a script which syncs the host time to the guest using qemu-guest-agent:

#!/bin/sh

time_ns=$(/bin/date +%s)000000000
echo "{\"execute\": \"guest-set-time\", \"arguments\":{\"time\": $time_ns}}" | /usr/bin/nc -U /Users/teuf/.crc/tap.sock

A similar approach could be used with qemu.

This is being worked on in https://github.com/cfergeau/vfkit/commits/timesync

[cfergeau]

@gbraad pointed me at https://github.com/prashantgupta24/mac-sleep-notifier which should be much better than sleepwatcher as it's native go/cgo code so it can directly be reused in crc/podman machine/...

[cfergeau]

The host side code for time sync for vfkit/crc is https://github.com/cfergeau/vfkit/blob/timesync/cmd/vfkit/timesync.go This should be fairly easy to adapt for use with qemu (replace the call to vf.ConnectVsockSync with something connecting over virtio-serial)

[ashley-cui]

Alright, I'm starting to actively work on this, thanks for holding tight y'all :)

[ssbarnea]

@ashley-cui Thanks. Ping me for testing, macos support is important for me and I would be more than happy to spend some time and help making podman on macos easier to use.

[xRomZak]

Got this issue with Minio container (after Mac sleep/awaik). Minio rejects requests when there is a time diff between client and server.

ErrorResponse(
  code = RequestTimeTooSkewed,
  message = The difference between the request time and the server's time is too large.
)

[stevenharrisBG]

There is a long blog post about this on the Docker blog. https://www.docker.com/blog/addressing-time-drift-in-docker-desktop-for-mac/

Docker Desktop runs an embedded NTP server on the host. It takes his source on the system time. A NTP client in the VM keeps it in sync.

I'm not certain these are the same issues - though they could be, given other variables.

I did not have time drift in my postgres containers using Docker. Using rootless podman, it is a very serious problem related to power saving and sleep, as far as I can tell.

[jamesmortensen]

Here's another method I came across to reset the clock manually. I wasn't able to use one of the other methods, so I did this. I stopped the ntp service (which didn't seem to be working anyway) and then set the time manually:

# SSH into the VM
$ podman machine ssh

# From within the VM:
$ sudo timedatectl set-ntp false
$ sudo timedatectl set-time "10:02”
$ exit

[jordansissel]

I ran into this and used this as a workaround:

podman machine ssh 'sudo hwclock -s'

[sc68cal]

I got hit with this recently when trying to push a container to an internal registry and SSL failed because the clock in the podman VM on MacOS was a couple weeks behind.

Rebooting it fixed the issue but we should try and get the VM to stay in sync timewise

[crahan]

I installed sleepwatcher via brew and then added the following to my ~/.wakeup script to update the date and time whenever my Mac wakes up:

#!/bin/bash

# Fix Podman Machine's time drift on wake
if [[ $(/usr/local/bin/podman machine info) ]]; then
    the_date=$(date +'%Y-%m-%dT%H:%M:%S')
    /usr/local/bin/podman machine ssh sudo date --set $the_date
    echo "Updated Podman Machine date: $the_date" >> ~/podman_debug.log
fi

So far, so good (until a long term fix becomes available).

[simon-friedberger]

I edited /etc/chrony.conf and added maxchange 100000 1 2 which lets chrony fix the time when it is very off. (Disclaimer: Not properly researched. Just seems to work for now.)

[fnkr]

makestep seems to fit this use-case exactly:

in some cases (e.g. a computer without an RTC or virtual machine which can be suspended and resumed with an incorrect time) it might be necessary to allow the step on any clock update. The example above would change to

makestep 1 -1

Source

To update the default Podman-Machine:

podman machine ssh --username root -- sed -i 's/^makestep\ .*$/makestep\ 1\ -1/' /etc/chrony.conf
podman machine ssh --username root -- systemctl restart chronyd

[xordspar0]

Makestep does indeed seem like the right solution. It looks like makestep 1 -1 was discussed in a Fedora CoreOS bug and the conclusion was that it should be the default.

I'm not sure why that doesn't show up in our images. Oh, I see. The change spawned by that discussion is only for cloud VM images. Either we should make Podman do the same for its image or we should bring this up in the Fedora CoreOS project.

[stephennimmo]

This is still an issue. I ran into it as I was using Keycloak as a Quarkus devservices container and the tokens issued were using the VM time and therefore already expired upon issuance.

[mrkvh]

Same issue here, noticed today after my mac hibernated over the weekend. Nothing helped except restarting the machine.

[xordspar0]

Either we should make Podman do the same for its image or we should bring this up in the Fedora CoreOS project.

I asked Fedora CoreOS if it would make sense to use makestep 1 -1 as the default, but it doesn't appear to be a good default for all qemu VMs. In that case I propose that we add this configuration to the default Podman machine image.

[baude]

it could easily be added to the ignition file that is used to configure the coreos instance.

[xordspar0]

it could easily be added to the ignition file that is used to configure the coreos instance.

Ok, I wondered if Podman might already had its own Ignition configuration. I found it and did just that in #17661.

[jgresham]

it could easily be added to the ignition file that is used to configure the coreos instance.

Ok, I wondered if Podman might already had its own Ignition configuration. I found it and did just that in #17661.

Thanks for the PR. I have been experiencing this issue numerous times and this seems like a good fix! Hoping it goes thru soon.