Schedules running an hour early, and then every minute until restart

sprocketaudio commented 1 year ago

Current Behaviour

Since the clocks changed here in the UK, my schedules that run once per 4 hours, are now running an hour early - this isnt really the issue though.. The problem is once they run they then run over and over again every minute until a service restart.

I have been trying to fix this for a couple weeks, and came across this - https://github.com/pterodactyl/panel/issues/2596

So to try and make sure everything was set correctly, I edited (what I believe) are all the places I can set the time zone.

These are: Maria Db current timezones MySQL config files Pterodactyl config files config.yml and .env

I have included pictures of all of these. I have done a physical server restart since then and the same issue happens - all schedules run early then repeat every 1 minute. I have searched discord and asked many times but do not get very helpful responses.. I also see many people seem to have reported this recently!

If I have done this wrong, or missed a config, please let me know.. otherwise I wonder if this old bug stil exists in some capacity?

Expected Behaviour

The schedules should run on time and with the correct time in between (like they did before the clocks changed for winter)

Steps to Reproduce

Not sure, but I would guess having Mysql date set different from panel date, then having the clocks change. After that I do not seem to be able to rectify the issue

Panel Version

1.10.1

Wings Version

1.7.0

Games and/or Eggs Affected

Minecraft

Docker Image

No response

Error Logs

None

Is there an existing issue for this?

[X] I have searched the existing issues before opening this issue.
[X] I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
[X] I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

Boy132 commented 1 year ago

Can you show your schedule? Sounds like you set the minute to *.

sprocketaudio commented 1 year ago

yeah - I appreciate you need to check, but i havent changed anything and have 2 servers running restart, start, backup and save schedules for 4 months with no problems. I'll post my save-all schedule and a pic of the console.

As i said this atarted on the 29/10/22 which is the night the clocks changed

parkervcp commented 1 year ago

UTC doesn't have daylight savings and stays constant. If the host your panel is running on is in UTC then that would be your issue.

I believe the current command to check is timedatectl

sprocketaudio commented 1 year ago

ah sorry I should have also ss this - this has been set since i first set the server up as Europe/London.

sprocketaudio commented 1 year ago

Any more ideas on this please? I can see this has been raised multiple times on both here and discord with no real answers.

https://github.com/pterodactyl/panel/issues/2075

https://github.com/pterodactyl/panel/issues/3351

I checked the code edits after https://github.com/pterodactyl/panel/issues/2596 was logged and i'm trying to understand it.. https://github.com/pterodactyl/panel/commit/996fb5b46f6ebaa9e38d58d3aeeb2feae654e814

Is Ptera setting the SQL timezone in the connection string?

Right now I have tried changing everything to UTC but still same issue.

System time - timedatectl - Time zone: UTC (UTC, +0000)

PHP time - date.timezone = Etc/UTC

Panel timezone(/var/www/pterodactyl/.env) : APP_TIMEZONE=Etc/UTC

Wings timezone(/etc/pterodactyl/config.yml) : timezone: Etc/UTC

SQL :

I have restarted the host server and recreated the 10 minutes save schedule and this is the console log:

parkervcp commented 1 year ago

Do you have multiple schedulers running?

sprocketaudio commented 1 year ago

I have only installed wings/panel once and have 2 Minecraft servers running on it.. so no i do not think so.

After running

sudo systemctl list-units --type service

I have 1 pteroq.service and 1 wings.service running

EDIT - I'm very close to just rolling a new host and installing OS and Ptera all fresh keeping all time zones default (which i think would make them all UTC) as the issue seems to effect people that have changed it that experience the clocks going forward or back. - tried to make mine Europe/London originally..

themikeambrose commented 11 months ago

@sprocketaudio Ever find a solution besides simply deleting everything and reinstalling? Also having this exact issue after DST change. Changed everything (sql, php, OS) to UTC and various time zone changes and nothing helps.

sprocketaudio commented 11 months ago

I didnt no - I reinstalled it all and set all times to UST as this never changes! Sorry

Boy132 commented 11 months ago

@sprocketaudio Ever find a solution besides simply deleting everything and reinstalling? Also having this exact issue after DST change. Changed everything (sql, php, OS) to UTC and various time zone changes and nothing helps.

Try panel version 1.11.4 or newer. There was a timezone related change in this version that maybe can help.

ecocraft commented 11 months ago

Same issue here on panel 1.11.5 Timezone: Europe/Prague

everytime I do php /var/www/pterodactyl/artisan schedule:run it runs the schedules ~~no matter the cron expression~~ (0 */1 * * *) it respects the cron expression (* * * */1 0) I was wondering if it was flipped since in the db its stored from right to left (probably not the case) But i see that its storing the time in a UTC time not the server/timezone time maybe thats the issue?

So i just tried to edit the time values (add 1 hour to match the timezone) in the db manually and the cron expression */5 * * * * doesnt run at 12:09 (tested again after and doesnt run on 12:13 and 12:14) now i edited the schedule so it saves the "wrong" time again, and the cron expression */5 * * * * runs at 12:11 and 12:12

I will try to tinker with it, but now i am like 80% certain that it messes with the time in the db.

My reasoning is that it sets the wrong time in the database (UTC) and later compares the time when it should run, since im using the Prague timezone which is +1 hour, it sees that the schedule is 1 hour behind and runs it no matter the set cron expression, and sets the current UTC time back in the db.

I hope this could help in some way.

Let me know how that goes if someone tries it.

Edit: added code brackets to the cron expressions so it doesnt break the formatting

themikeambrose commented 11 months ago

@sprocketaudio Ever find a solution besides simply deleting everything and reinstalling? Also having this exact issue after DST change. Changed everything (sql, php, OS) to UTC and various time zone changes and nothing helps.

Try panel version 1.11.4 or newer. There was a timezone related change in this version that maybe can help.

It was tested, 1.11.4 panel does not fix the schedule issue.

themikeambrose commented 10 months ago

If anyone is able to help find and document the cause and fix without wiping all data contact me. Will pay for the time and solution, since this issue has affected many people over the few years it seems and there's never a real fix.

Edit 12-23-2023 Ended up reinstalling like everyone else does. Reinstalled to Debian 12 (from Almalinux 9) and imported my old .env and database. After messing with the new panel install I have pteroq/tasks running properly without issue. I'll see if it lasts, but perhaps the issue might be isolated to Almalinux/RHEL installs or something? Maybe differences with something like redis causing problems? No idea..

KenRen98 commented 8 months ago

any fix?

Boy132 commented 8 months ago

For anyone with the same problem, please provide the following information:

What OS are you using?
What Panel and Wings version are you using?
Are you running the dockerized versions of the panel and/ or wings? If yes which images are you using?
Show the output of timedatectl
Show the status of the queue worker (systemctl status pteroq)

Run the following commands and show the output:

cd /var/www/pterodactyl
php artisan tinker
config('app.timezone')
config('database.connections.mysql.timezone')
exit

(Optional) A screenshot of your schedule where the "Last Run", "Next Run" and time fields are visible

KenRen98 commented 8 months ago

For anyone with the same problem, please provide the following information:

What OS are you using?

What Panel and Wings version are you using?

Show the output of timedatectl

Show the status of the queue worker (systemctl status pteroq)

Run the following commands and show the output:
cd /var/www/pterodactyl
php artisan tinker
config('app.timezone')
config('database.connections.mysql.timezone')
(Optional) A screenshot of your schedule where the "Last Run", "Next Run" and time fields are visible

Nvm, resolved by setting all system time and App timezone and wings timezone to Etc/Universal. Then rebooted panel.

KenRen98 commented 8 months ago

This would solve both problems,

Log showing future time
Scheduler execute every minute and ignoring the interval

pterodactyl / panel