fmi-faim / prefect-workflows

A collection of Prefect workflows.
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Provide DaskRunner configuration(s) via block #18

Open imagejan opened 1 year ago

imagejan commented 1 year ago

Can we provide configuration like this one:

https://github.com/fmi-faim/prefect-workflows/blob/69c4c41e74425d55127e47c9868a97615d042f01/eicm_flows/compute_all_matrices.py#L46-L73

via a block, so we avoid

tibuch commented 1 year ago

Parts of it probably yes, but last I tried I could not reliably retrieve the data from blocks outside of the flow.

imagejan commented 1 year ago

This worked for me (with prefect==2.7.7):

from prefect.blocks.system import JSON

test_kwargs = JSON.load('test-kwargs').value

@flow(
    **test_kwargs
)
def test_flow():
    get_run_logger().info("Running test flow...")

... where the test-kwargs block provided a name and description attribute for the flow annotation. (I first tested a custom block class, before noticing that the JSON block provided by prefect serves our purpose just right.) Tested on the faim-dev workspace.