Timezone discrepancy between container and host

instantdreams commented 3 months ago

Support guidelines

[x] I've read the support guidelines

I've found a bug and checked that ...

[x] ... the documentation does not mention anything about my problem
[x] ... there are no open or closed issues that are related to my problem

Description

Date on host reports according to timezone. Date on container reports in UTC.

Expected behaviour

For timezone to be used by container.

Actual behaviour

Container date command is reporting UTC.

Steps to reproduce

Enable and set docker environment variable TZ
Add read only volume mount for /etc/timezone
Add read only volume mount for /etc/localtime
Start diun container
Run command timedatectl
Run command docker exec diun-1 timedatectl
Run command date
Run command docker exec diun-1 date

Diun version

4.26.0

Docker info

Client: Docker Engine - Community
 Version:    25.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 10
 Server Version: 25.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.21-v8+
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.855GiB
 Name: id-edge1
 ID: 7HKR:XS7T:KP55:WXII:JDYT:SCR5:EF3N:PHZA:XOAK:EWEP:3J7W:NWFN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker Compose config

name: diun
networks:
  default:
    name: diun_default
  example-net:
    external: true
    name: example
services:
  diun:
    command: serve
    container_name: diun-1
    domainname: example.com
    environment:
      - TZ=America/Edmonton
      - LOG_LEVEL=info
      - LOG_JSON=false
      - DIUN_WATCH_WORKERS=5
      - DIUN_WATCH_SCHEDULE=10 06 * * 0
      - DIUN_WATCH_JITTER=30s
      - DIUN_WATCH_FIRSTCHECKNOTIF=true
      - DIUN_PROVIDERS_DOCKER=true
      - DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true
      - DIUN_PROVIDERS_DOCKER_WATCHSTOPPED=true
      - DIUN_NOTIF_TELEGRAM_TOKEN=[token]
      - DIUN_NOTIF_TELEGRAM_CHATIDS=[chat-id]
    hostname: diun-1
    image: ghcr.io/crazy-max/diun:latest
    labels:
      - traefik.enable=false
    networks:
      example-net:
        ipv4_address: 192.168.91.231
    restart: unless-stopped
    volumes:
      - bind:
          create_host_path: true
        source: /srv/diun/data
        target: /data
        type: bind
      - bind:
          create_host_path: true
        read_only: true
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        type: bind
      - bind:
          create_host_path: true
        read_only: true
        source: /etc/timezone
        target: /etc/timezone
        type: bind
      - bind:
          create_host_path: true
        read_only: true
        source: /etc/localtime
        target: /etc/localtime
        type: bind

Logs

diun-1  | Wed, 20 Mar 2024 09:46:53 MDT INF Starting Diun version=v4.26.0
diun-1  | Wed, 20 Mar 2024 09:46:53 MDT INF Configuration loaded from 10 environment variable(s)
diun-1  | Wed, 20 Mar 2024 09:46:53 MDT INF Cron triggered
diun-1  | Wed, 20 Mar 2024 09:46:54 MDT INF Found 10 image(s) to analyze provider=docker
diun-1  | Wed, 20 Mar 2024 09:46:55 MDT INF Jobs completed added=0 failed=0 skipped=0 unchanged=10 updated=0
diun-1  | Wed, 20 Mar 2024 09:46:55 MDT INF Cron initialized with schedule 10 06 * * 0
diun-1  | Wed, 20 Mar 2024 09:46:55 MDT INF Next run in 3 days 20 hours (2024-03-24 06:10:03.337658867 -0600 MDT)

Additional info

Here's the output:

$ timedatectl
               Local time: Wed 2024-03-20 10:12:46 MDT
           Universal time: Wed 2024-03-20 16:12:46 UTC
                 RTC time: n/a
                Time zone: America/Edmonton (MDT, -0600)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

$ docker exec diun-1 timedatectl
OCI runtime exec failed: exec failed: unable to start container process: exec: "timedatectl": executable file not found in $PATH: unknown

$ date
Wed Mar 20 10:13:07 MDT 2024

$ docker exec diun-1 date
Wed Mar 20 16:13:10 UTC 2024

I am expecting the result of docker exec diun-1 date to match the local container.

Fayyaadh commented 3 months ago

I think I might be having the same issue.

I updated the diun container to the latest image and where I used to have it check everyday at 2pm (UTC+2), today it decided to check at 4pm.

root@portainer:~# date
Mon Mar 25 16:27:03 SAST 2024
root@portainer:~# docker exec diun date
Mon Mar 25 14:27:34 UTC 2024

Guiorgy commented 3 months ago

This is a general problem with containers based on Alpine Linux. By default the image doesn't come with the tzdata package preinstalled, which is necessary for time zones to work properly. You can try this (replace Diun with the container name/id if different):

# docker exec -it Diun exec ash
# date
Mon Mar 25 **:**:** UTC 2024
# apk add --no-cache tzdata
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tzdata (2024a-r0)
OK: 12 MiB in 18 packages
# date
Mon Mar 25 **:**:** +** 2024

You can read this issue if you want to know why the package is not included with the image.

Unless Diun requires time zones to work properly, they are an optional, so the user should install the package if desired.

ch4ox commented 3 months ago

I had the same problem since the v4.27.0 update -> timezone Europe/Berlin [UTC+1] for me.

What helped for me:

removing the TZ environment variable (<- this is important)
using "/etc/localtime:/etc/localtime:ro" as a volume mount

Hope this still works after daylight saving next weekend.

Fayyaadh commented 3 months ago

Everything was fine prior to updating diun a few days ago to the latest image.

Guiorgy commented 3 months ago

Everything was fine prior to updating diun a few days ago to the latest image.

If that's the case, first of all, didn't notice, and secondly, the only significant differences in the dockerfile I can see are version bumps for golang (1.20 -> 1.21) and tonistiigi/xx (1.2.1 -> 1.3.0), of which the later had substantially more refactoring done, but I didn't see tzdata referenced directly, nor anything that depends on it directly, so perhaps it was a transient package?

Fayyaadh commented 3 months ago

Everything was fine prior to updating diun a few days ago to the latest image.

If that's the case, first of all, didn't notice, and secondly, the only significant differences in the dockerfile I can see are version bumps for golang (1.20 -> 1.21) and tonistiigi/xx (1.2.1 -> 1.3.0), of which the later had substantially more refactoring done, but I didn't see tzdata referenced directly, nor anything that depends on it directly, so perhaps it was a transient package?

No idea.

But no config has changed on my side at all, and this started happening after updating to the latest release. I also had a look at the release notes and nothing obvious stood out.

I guess when the developer sees this thread, he'll look into it.

Apart from that, I guess I can try the suggestion above to get rid of the environment variable and get local time from the host. Or the more hacky way of adjusting my cron schedule so that's it's UTC.

Jefe2 commented 3 months ago

I’m also having the same problem, ever since updating to v4.27.0 a couple days ago.

crazy-max commented 3 months ago

Thanks for reporting, I'm taking a look.

Guiorgy commented 3 months ago

# docker run -it --rm --platform linux/amd64 --entrypoint ash crazymax/diun:4.26
/ # apk info -vv
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.18/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.18/community: No such file or directory
alpine-baselayout-3.4.3-r1 - Alpine base dir structure and init scripts
alpine-baselayout-data-3.4.3-r1 - Alpine base dir structure and init scripts
alpine-keys-2.4-r1 - Public keys for Alpine Linux packages
apk-tools-2.14.0-r2 - Alpine Package Keeper - package manager for alpine
busybox-1.36.1-r2 - Size optimized toolbox of many common UNIX utilities
busybox-binsh-1.36.1-r2 - busybox ash /bin/sh
ca-certificates-20230506-r0 - Common CA certificates PEM files from Mozilla
ca-certificates-bundle-20230506-r0 - Pre generated bundle of Mozilla certificates
libc-utils-0.7.2-r5 - Meta package to pull in correct libc
libcrypto3-3.1.3-r0 - Crypto library from openssl
libssl3-3.1.3-r0 - SSL shared libraries
musl-1.2.4-r1 - the musl c library (libc) implementation
musl-utils-1.2.4-r1 - the musl c library (libc) implementation
openssl-3.1.3-r0 - Toolkit for Transport Layer Security (TLS)
scanelf-1.3.7-r1 - Scan ELF binaries for stuff
ssl_client-1.36.1-r2 - EXternal ssl_client for busybox wget
zlib-1.2.13-r1 - A compression/decompression Library

Doesn't look like tzdata was ever installed... 🤔

jicho commented 3 months ago

Same here...

I have TZ=Europe/Amsterdam in my docker compose in combination with DIUN_WATCH_SCHEDULE=0 7 * * *

Result is since the latest release that the scheduler runs at 8 local time... Before the update the scheduler runs, as expected, at 7.

crazy-max commented 3 months ago

I'm able to repro:

t := time.Now()
zone, offset := t.Zone()
fmt.Println("TZ", os.Getenv("TZ"))
fmt.Println("zone", zone)
fmt.Println("offset", offset)

diun  | TZ Europe/Paris
diun  | zone UTC
diun  | offset 0

Might be a change since Go 1.21.

Doesn't look like tzdata was ever installed... 🤔

@Guiorgy This is not needed, we already embed tzdata in the app: https://github.com/crazy-max/diun/blob/756752964d28b673b46d19e0fdc55fc9fe156dbb/cmd/main.go#L8

crazy-max commented 3 months ago

With Go 1.20 it works as expected:

diun  | TZ Europe/Paris
diun  | zone CET
diun  | offset 3600

crazy-max commented 3 months ago

Looks related to this change https://github.com/golang/go/commit/627765a861c6762d15cfdfeb2f7e4872523edbe4 in Go 1.21.

Installing tzdata package in the image seems to work so this is an issue with embedded tzdata from Go.

Fayyaadh commented 3 months ago

Looks related to this change https://github.com/golang/go/commit/627765a861c6762d15cfdfeb2f7e4872523edbe4 in Go 1.21.

Installing tzdata package in the image seems to work so this is an issue with embedded tzdata from Go.

So will there be an updated image with a fix, or how do end users deal with the issue?

Guiorgy commented 3 months ago

@Fayyaadh As mentioned, installing tzdata manually is a temporary solution. Just run the following (replace Diun with the name of your container):

docker exec -it Diun apk add --no-cache tzdata

After it's installed, restart the container:

docker restart Diun

Fayyaadh commented 3 months ago

@Guiorgy thank you, sir, I've done that now and will see if the notification goes out at the correct time.

Guiorgy commented 3 months ago

@Guiorgy thank you, sir, I've done that now and will see if the notification goes out at the correct time.

Forgot to mention 1 thing, you'll probably need to restart (not recreate) the container for the timezone to apply, for example my logs:

Wed, 27 Mar 2024 10:23:09 UTC INF Next run in 4 hours 37 minutes (2024-03-27 15:00:14.30801095 +0000 UTC)
Wed, 27 Mar 2024 10:23:20 UTC WRN Caught signal terminated
Wed, 27 Mar 2024 14:23:23 +04 INF Starting Diun version=v4.27.0
...
Wed, 27 Mar 2024 14:23:27 +04 INF Cron initialized with schedule 0 15 * * *
Wed, 27 Mar 2024 14:23:27 +04 INF Next run in 36 minutes 41 seconds (2024-03-27 15:00:09.353141199 +0400 +04)

Fayyaadh commented 3 months ago

@Guiorgy thank you, sir, I've done that now and will see if the notification goes out at the correct time.

Forgot to mention 1 thing, you'll probably need to restart (not recreate) the container for the timezone to apply, for example my logs:
Wed, 27 Mar 2024 10:23:09 UTC INF Next run in 4 hours 37 minutes (2024-03-27 15:00:14.30801095 +0000 UTC)

Wed, 27 Mar 2024 10:23:20 UTC WRN Caught signal terminated

Wed, 27 Mar 2024 14:23:23 +04 INF Starting Diun version=v4.27.0

...

Wed, 27 Mar 2024 14:23:27 +04 INF Cron initialized with schedule 0 15 * * *

Wed, 27 Mar 2024 14:23:27 +04 INF Next run in 36 minutes 41 seconds (2024-03-27 15:00:09.353141199 +0400 +04)

I think it worked without restarting the container. Running the date command inside the container now shows my local timezone, not UTC anymore.

kristof-mattei commented 3 months ago

@Guiorgy thank you, sir, I've done that now and will see if the notification goes out at the correct time.

Forgot to mention 1 thing, you'll probably need to restart (not recreate) the container for the timezone to apply, for example my logs:
Wed, 27 Mar 2024 10:23:09 UTC INF Next run in 4 hours 37 minutes (2024-03-27 15:00:14.30801095 +0000 UTC)

Wed, 27 Mar 2024 10:23:20 UTC WRN Caught signal terminated

Wed, 27 Mar 2024 14:23:23 +04 INF Starting Diun version=v4.27.0

...

Wed, 27 Mar 2024 14:23:27 +04 INF Cron initialized with schedule 0 15 * * *

Wed, 27 Mar 2024 14:23:27 +04 INF Next run in 36 minutes 41 seconds (2024-03-27 15:00:09.353141199 +0400 +04)
I think it worked without restarting the container. Running the date command inside the container now shows my local timezone, not UTC anymore.

It never had the tzdata installed, so while date might pick it up, it is a new process after all, I'm not sure the go process picks it up.

Guiorgy commented 3 months ago

@kristof-mattei The cron was triggered correctly, so it works in this case:

DIUN_WATCH_SCHEDULE: '0 15 * * *'

2024-03-28T11:00:53 Thu, 28 Mar 2024 15:00:53 +04 INF Cron triggered

kradNZ commented 2 months ago

Same issue here. Thanks for the temporary fix.

crazy-max / diun