Prefect v2 to v3 transition, idempotency key generation not consistent

[mattijsdp]

Bug summary

I recently upgraded to Prefect v3. I have some runs which run daily and Prefect creates scheduled runs such that there are always 3 in the future. With the transition, these daily runs accidentally ran twice for three days and then ran once again as expected.

I believe this is because the idempotency key generation is every so slightly different e.g.: scheduled d6c42edb-55da-4dc0-9a62-b9cb693695e4 2024-09-22T01:00:00+00:00 vs. scheduled d6c42edb-55da-4dc0-9a62-b9cb693695e4 2024-09-22 01:00:00+00:00 Notice the space after the date instead of 'T'. It is quite hard for me to reproduce this unfortunately but I thought it would be good to share.

Version info (prefect version output)

The output post upgrade:

Version:             3.0.2
API version:         0.8.4
Python version:      3.11.9
Git commit:          c846de02
Built:               Fri, Sep 13, 2024 10:48 AM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.9.1

### Additional context

_No response_

[zzstoatzz]

hi @mattijsdp - thank you for the issue! we will look into this

---

for later, here's where this is coming from

and it appears the difference is that we are now using the default pendulum datetime -> str render, as opposed to iso8601

In [1]: from pendulum import DateTime

In [2]: f"{DateTime.now()}"
Out[2]: '2024-09-24 14:53:26.419160-05:00'

In [3]: f"{DateTime.now().to_iso8601_string()}"
Out[3]: '2024-09-24T14:53:30.409176-05:00'