search

hiddeco / cronjobber

Cronjobber is a cronjob controller for Kubernetes with support for time zones
Apache License 2.0
244 stars 38 forks source link

Cronjobber does not work with slim tzdata format #34

Open scorsi opened 3 years ago

scorsi commented 3 years ago

Hello,

I have a two CronJob the one set to 0 1 * * * in Europe/Paris which is running at 3 CEST, the other one scheduled at 0 3 * * * in Europe/Paris ran at 5 CEST...

Do you have any idea why it's not running at the time it was scheduled ?

Thanks,

u6f6o commented 3 years ago

Hello,

Same issue here. We normally scale up our pods in the evening, yet they have been triggered in the morning today. I found the following text in the logs: 

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Best, u6f6o

pscheid92 commented 3 years ago

Hello, @u6f6o and I have the exact same problem.

We have 4 different namespaces with 2 jobs each. Of those jobs, 4 are scheduled for 19:50 PM and 4 are scheduled for 20:30 PM, each day.

Today, they all ran at exactly the same time (10:44 AM). Maybe it is a coincidence, but the updatetz container logged the right before all jobs ran:

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Our local time zone is CEST (+0200), thus the database update was logged at 10:43:58 AM local time.

scorsi commented 3 years ago

May be related to #24. Actually don't have the workload to investigate deeper. Switched to default Kubernetes CronJob while the issue is not fixed/investigated.

obitech commented 3 years ago

Running into the same problem here. Now my TZCronJobs won't start at all, at least not during my specified time. I haven't been able to debug the tzdb inside the container, has there been an issue with the update?

hiddeco commented 3 years ago

Sorry about this unexpected behaviour, and I hope it did not result in too much chaos. Can you all please confirm you are using the latest cronjobber-updatetz release (v0.1.1)? This helps me narrow my search.

pscheid92 commented 3 years ago

Yes, we (@u6f6o, @obitech and me) used the v0.1.1 release (with sidecar.enabled=true; installed per Helm chart).

apicht commented 3 years ago

Just noticed this is happening for us as well. Running cronjobber:0.2.0 and cronjobber-updatetz:0.1.0. Latest log message from updatetz is

2020-10-22T23:03:54+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo
maksimu commented 3 years ago

Yeap, same for me. Hopefully it can be fixed soon

maksimu commented 3 years ago

When I look at the logs of the cronjobber I see this errors that keep repeating over and over again:

2020-10-25T07:09:06.312819787Z cronjobber E1025 07:09:06.312568       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope
2020-10-25T07:09:07.182842095Z cronjobber E1025 07:09:07.182647       1 controller.go:122] can't list Jobs: jobs.batch is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "jobs" in API group "batch" at the cluster scope
2020-10-25T07:09:07.316420688Z cronjobber E1025 07:09:07.316221       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope

Maybe something is wrong w/ the service account? In fact my service account that was installed before was gone.

obitech commented 3 years ago

@maksimu Your errors should be unrelated. Your service account seems to be missing the correct ClusterRole permissions.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: cronjobber
rules:
- apiGroups: ['cronjobber.hidde.co']
  resources: ['*']
  verbs: ['*']
- apiGroups: ['batch']
  resources: ['jobs']
  verbs: ['*']
- apiGroups: ['']
  resources: ['events']
  verbs: ['create', 'patch', 'update']
hiddeco commented 3 years ago

The problem is due to zic changing the default format from fat to slim since >=2020-b, this format is however not correctly understood by Go (https://github.com/golang/go/issues/42138), resulting in an UTC fallback.

The best option here is likely to start using Go 1.15, and make use of the embedded timezone information there, which makes the timezone update utility obsolete.

obitech commented 3 years ago

@hiddeco do you find time to fix this? Otherwise I can draft a PR.

hiddeco commented 3 years ago

I have just crafted a release, once it has been pushed to the registry, using the https://raw.githubusercontent.com/hiddeco/cronjobber/master/deploy/deploy.yaml manifest with quay.io/hiddeco/cronjobber:0.3.0 should solve this issue with a workaround.

I will track the Go issue to release a new version once it is able to understand the slim format as well, so the updatetz utility can be used again. Keeping this issue open until then.

obitech commented 3 years ago

Works again with 0.3.0, thank you! 🙏

hiddeco commented 3 years ago

The bug fix for this problem has landed in the release branch for Go 1.15.4, with the release scheduled for next week. Expect a patch release as soon as this version is available in container / image land. :sunflower:

rmak-cpi commented 3 years ago

I am just wondering when cronjobber is going to be patched with the golang fix (starting from 1.15.4) for handling slim tzdata given that we are already past the 2021 DST transition. Thanks!

hiddeco commented 3 years ago

Hello all,

It is likely not what you hoped, but I do at present not have the bandwidth to further maintain this project. See https://github.com/hiddeco/cronjobber/issues/43#issuecomment-890097573 for more information.