PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.32k stars 1.5k forks source link

Allow users to select and invoke flow runs with Blocks when using Block as a param type hint #14393

Open jcbiddle opened 3 days ago

jcbiddle commented 3 days ago

First check

Bug summary

I have a flow that uses different blocks to connect to different database endpoints and want the block name to be a parameter of the flow. If I use a string parameter for the block name and load the existing block from this, the code works as expected. However, to provide additional clarity I want the Prefect UI to give a dropdown list of existing blocks, rather than require the user to recall the string identifier.

I found that adding the block class as a type hint causes the Prefect UI to do exactly what I want, giving a dropdown menu for existing blocks of the correct type, with functionality to add a new block. However, attempting to execute the run after selecting a block doesn't work. No error is provided in the form input view, however, in the JSON view it is clear that the parameter JSON expects the block parameters to be provided, even if the block is a reference to an existing block with the parameters pre-configured.

To replicate this issue using the MWE provided, first register CustomBlock with: prefect block register --file mwe.py Then load the existing block and deploy the flow with python mwe.py

The run deployment menu appears as image which seems to be exactly the desired behaviour. However, if we run the flow from the JSON parameter screen, we get the following error: image

I assume the existing block parameters are supposed to be loaded into the run parameters, but this doesn't seem to be working as intended.

As a side-note, there also seems to be a bug where if the user switches to the JSON parameter view, the button to switch back to the form view doesn't work and requires a page refresh instead.

Reproduction

from prefect import flow
from prefect.blocks.core import Block

class CustomBlock(Block):
    example_field: str

@flow
def my_flow(block: CustomBlock):
    b = CustomBlock.load(block)
    print(b.example_field)

if __name__ == "__main__":
    my_block = CustomBlock(example_field="hello world")
    my_block.save("my-block")

    my_flow.serve("flow-deployment")

Error

- block
- example_field
- 'example_field' is a required property

Versions (prefect version output)

Version:             3.0.0rc9
API version:         0.8.4
Python version:      3.11.9
Git commit:          b75631a7
Built:               Mon, Jun 24, 2024 5:50 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Pydantic version:    2.7.4
Server:
  Database:          sqlite
  SQLite version:    3.37.2

Additional context

No response

WillRaphaelson commented 2 days ago

Thanks for the issue @jcbiddle, we can look at supporting this in the near future. Variables should work as you expect here in terms of the dropdown user experience, and then you can use the variable value in your block.load() call.

jcbiddle commented 17 hours ago

Thanks for having a look at this @WillRaphaelson. I'm new to Prefect, can you possibly expand on how I can use variables in this context to get a dropdown menu in the UI? I tried adding Variable as a type hint for the flow parameter, but that just gave me the option to make a new variable rather than select an existing one.

WillRaphaelson commented 1 hour ago

Hey @jcbiddle, for sure.

The type hint can be anything, in this case I lets use str. When you go to run the flow from the deployment, hit the three dots and use select variable. Then you'll get your dropdown. The value of the variable will then fall through to your flow run, and you can use that to retrieve the appropriate block with block.load(variable_value). This does require you maintain a list of variables in congruence with your blocks (or at least the ones that you want available for use in the flow). this is a workaround, and im going to keep this issue open with an edited title to reflect the feature request of allowing users to select blocks as a dropdown and invoke flows with them without erroring out :)

image