PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.82k stars 1.55k forks source link

Schedules missing in deployment page #14618

Open DataOps7 opened 1 month ago

DataOps7 commented 1 month ago

First check

Bug summary

I'm getting this error on Prefect community edition version 3.0.0rc10 When looking at a specific deployment page, at the top right, the "Schedules" title is there but the schedules are missing.

When going to Chrome's dev console (f12), I'm seeing the following error:

TypeError: a.schedules.toSorted is no a function

Can't upload a screenshot due to company policy.

Reproduction

Open any deployment page

Error

No response

Versions (prefect version output)

Version: 3.0.0rc10
API version: 0.8.4
Python version: 3.10.14
GitCommit: 200cf212
Built: Wed, Jul 3, 2024 1:39 PM
OS/Arch: linux/x86_64
Profile: default
Server type: server
Pydantic version: 2.8.0

Additional context

No response

aaazzam commented 1 month ago

🙇 — thanks for reporting. This feels like a high priority issue, and we'll address it accordingly.

zhen0 commented 1 month ago

Hi @DataOps7 - thanks for the issue! I appreciate that you can't share a screenshot but is it possible to share the schedule you are using? I can't reproduce with a simple interval or cron schedule so any extra information you can provide would be helpful. Thanks!

barskern commented 1 month ago

I have the same issue. See attached screenshot of the missing schedules below. Its worth noting that despite not showing the schedules, a schedule is configured through the CLI and the schedule is functional.

Versions:

Python: 3.9.7 Prefect: 2.16.9 Microsoft Edge: 99.0.1150.46 (Official build) (64-bit)

image

DataOps7 commented 3 weeks ago

@zhen0 @barskern That's exactly what I'm experiencing!

DataOps7 commented 3 weeks ago

Just noticed the deployment stats are missing too, I don't have the graphs show below: (Noting that I'm running in an offline air-gapped network) image

barskern commented 3 weeks ago

The machine I see the issue on is also "offline", i.e. no access to the internet. (Perhaps it is relevant)

dylanbhughes commented 3 weeks ago

Thanks for that @DataOps7 and @barskern -- would you mind sharing the schedules for the deployments you're seeing issues for? I'm having trouble recreating this bug (even offline)

DataOps7 commented 1 week ago

@dylanbhughes I'm using python to deploy my flow, using a basic cron schedule like "0 0 *"

dylanbhughes commented 1 week ago

Hey @DataOps7, I'm still struggling to reproduce this. Could you please provide a reproduction? Here's an example of what we're looking for:

Prefect Version Output

Version: 3.0.0rc10
API version: 0.8.4
Python version: 3.10.14
GitCommit: 200cf212
Built: Wed, Jul 3, 2024 1:39 PM
OS/Arch: linux/x86_64
Profile: default
Server type: server
Pydantic version: 2.8.0

Flow Example

from pathlib import Path
from time import sleep

from prefect import flow, task
from prefect.client.schemas.schedules import CronSchedule

def fetch_batch(i: int):
    # Simulate fetching a batch of data
    sleep(2)

@task
def fetch_in_batches(batches: int):
    for i in range(1, batches + 1):
        fetch_batch(i)

@flow
def etl(batches: int = 10):
    fetch_in_batches(batches)

if __name__ == "__main__":
    etl.from_source(
        source=str(Path(__file__).parent),
        entrypoint="duplicate_deployment_example.py:etl",
    ).deploy(
        name="etl",
        work_pool_name="my_work_pool",
        schedules=[
            {
                "schedule": CronSchedule(cron="0 0 * * *"),
            }
        ],
    )

In this example, I'm using a process work pool & worker.

Result

Screenshot 2024-09-06 at 10 38 15 AM
DataOps7 commented 5 days ago

It's a bit of a problem to provide a reproduction as the problem only happens in an air gapped offline network. @barskern Do you have a way of reproducing this and providing the awesome Prefect team with the details?

I can try copying my flow line by line but it's just something like:

@flow
def my_flow():
    print("Hey!")
    return

if __name__ == "__main__":
    my_flow.deploy(cron="0 0 * * *")

The cron works fine, it's just a UI problem which also blocks me from pausing flows as the schedules are missing.

barskern commented 5 days ago

It's a bit of a problem to provide a reproduction as the problem only happens in an air gapped offline network.

@barskern Do you have a way of reproducing this and providing the awesome Prefect team with the details?

I can try copying my flow line by line but it's just something like:


@flow

def my_flow():

    print("Hey!")

    return

if __name__ == "__main__":

    my_flow.deploy(cron="0 0 * * *")

The cron works fine, it's just a UI problem which also blocks me from pausing flows as the schedules are missing.

Sadly no. I have seen the issue in at an airgapped system aswell. An the schedule itself doesnt matter. Further the exact same configuration and schedule is working when I run it on another server with internet access or on my laptop.

dylanbhughes commented 4 days ago

Thanks to both of you for sharing those details! I'm not very familiar with airgapped systems — can you tell me a little bit more about how you install Prefect onto those machines? I'm wondering if there's something about the installation process that's different from pip install prefect and if that results in misconfiguration or if there's an asset we're not properly including in the pypi distribution and is being fetched over the wire.

DataOps7 commented 2 days ago

Our Prefect server runs on K8S, and I used pip to install Prefect from a private pypi registry, it shouldn't be any different then online.

dylanbhughes commented 1 day ago

So it looks like there's some work pool information that we used to fetch from GitHub, but that's included in the PyPI package now. If you can share the full route that returns a 422, that could help us track this down.