PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.38k stars 1.59k forks source link

flow name doesn't change with `with_options` function #15380

Closed MeasureSpace closed 1 month ago

MeasureSpace commented 1 month ago

Bug summary

The flow name doesn't change with the with_option function. Even I changed the flow name with the with_option(name='xx'). The deployed flow name is still the default one. See following code to reproduce:

from prefect import flow
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials
from prefect.client.schemas.schedules import CronSchedule
import click

@click.command()
@click.option('--env', default='staging', help='Deployment environment: staging or prod.')
def deploy_flow(env):

    if env == 'staging':
        deployment_name = "staging-test"
        flow_name = "staging-test"
        branch = 'staging'
        paused = True
    elif env == 'prod':
        deployment_name = "prod-test"
        flow_name = "prod-test"
        branch = 'main'
        paused = False

    source = GitRepository(
        url="https://github.com/xx/xx.git",
        branch=branch,
        credentials=GitHubCredentials.load("aws-prefect"),
    )

    cron_schedule = CronSchedule(cron="0 0 * * *", timezone='UTC')
    entrypoint = "xx/xx/xx.py:main"

    flow.from_source(
        source=source,
        entrypoint=entrypoint,
    ).with_options(
        name=flow_name,
    ).deploy(
        name=deployment_name, 
        work_queue_name="xx",
        work_pool_name="default-agent-pool",
        schedules = [cron_schedule],
        paused=paused,
        build=False,
    )

if __name__ == '__main__':
    deploy_flow()

Version info (prefect version output)

Version:             3.0.1
API version:         0.8.4
Python version:      3.11.5
Git commit:          c6b2ffe1
Built:               Fri, Sep 6, 2024 10:05 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server
Pydantic version:    2.9.1
Integrations:
  prefect-github:    0.3.0

Additional context

No response

zzstoatzz commented 1 month ago

hi @MeasureSpace - hmm I'll note that this appears to work as expected

from prefect import __version__, flow

f = flow.from_source(
    source="https://gist.github.com/3eb9333625d465cb74381c5116be7aee.git",
    entrypoint="tour_of_artifacts.py:main",
).with_options(name="xx")

assert f.name == "xx"
assert __version__ == "3.0.1"

if __name__ == "__main__":
    f.serve()
image

can you explain where you're seeing the unexpected flow name?

MeasureSpace commented 1 month ago

Hi @zzstoatzz , thanks a lot for looking into this. I did the assert name as well. The name actually was changed. Now I believe the issue happened during the deploy() process. Please see the snapshot below. The deployment table shows GFS-data-ingestion/staging-gfs-realtime and the actual run command shows prefect deployment run 'staging-gfs-data-ingestion/staging-gfs-realtime'. In reality, the name remains as the default flow name which is GFS-data-ingestion. Any thoughts?

Screenshot 2024-09-14 at 8 57 22 AM
zzstoatzz commented 1 month ago

hi @MeasureSpace - I've added a PR to update the handling in .deploy