Flow runs duplicated when DST ends (v2)

[drfraser]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar issue and didn't find it.
• [X] I searched the Prefect documentation for this issue.
• [X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

A couple of days ago, I regenerated deployments, reset the database (it was getting too big) and flow runs were created and scheduled up to and on the 30th or so - I did not examine things as it has always worked without any issues.

The deployments are cron based and the timezone is set to Europe/London as is the server.

Today, after the end of BST, it seems duplicate flow runs were automatically generated around midnight for today (the 30th) based on the flow run attributes - i.e. the already existing ones for the 30th were ignored and new ones created that ran an hour later. As these flows were designed to run once a day, the 2nd ones were crashing. No data loss, given how the flows work, but annoying to see failed runs in the UI...

This of course is unexpected, though it makes some sense. IMO, the generation of flow runs process should take into account whether time has shifted to DST ending or starting and manage already existing runs that might be "equivalents" accordingly.

There were no problems with the one flow scheduled to run at an interval on every hour. Hopefully this is just a problem with my version of Prefect and upgrading to the latest will fix this, but I could not find any existing issues...

Reproduction

1. Create a flow + deployment and schedule the flow for every day at a specific time.
Do not use UTC as the timezone but one that uses DST - e.g. Europe/London

2. Wait until DST is over or starts, then see if Prefect has automatically created a duplicate flow run for the flow that is an hour later (or earlier)

Error

Flow runs scheduled to run only once a day were automatically duplicated upon the end of DST and caused problems.

Versions

Version:             2.0.4
API version:         0.8.0
Python version:      3.8.10
Git commit:          39db6fb1
Built:               Wed, Aug 10, 2022 1:19 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.31.1

Additional context

No response

[vsytchenko-exos]

Faced the same issue. Several pipelines were scheduled to run once a day with europe/warsaw timezone. 29-30th of October was a switch to winter time. One hour back. And I've noticed this:

[zanieb]

@zangell44 / @jlowin is this a bug with the scheduler idempotency keys or schedule itself?

[jlowin]

I think that exactly on the boundary it might be an idempotency key issue as the keys are being created from the raw date string here instead of a UTC-converted string (so two different time zones = two different date keys). However I’m confused about why that led to multiple duplicates instead of just one on the boundary — my hypothesis is that the scheduler created runs through the DST boundary with one TZ in the key, then created a second set with the other TZ.

I believe we have extensive tests for DST boundary behavior, especially with cron, so I’d have to see what’s different here or why this doesn’t affect them.

[drfraser]

I updated my Prefect setup to the latest version before the timezone change and did not see this problem when DST came around. So it's fixed as far as I care, but I will check to see if what jlowin mentions applies to my setup.