tonycombocurve
commented
2 years ago See the Prefect migration guide for further details:
https://docs.prefect.io/migration-guide/?h=executor#conceptual-and-syntax-changes
Closed tonycombocurve closed 2 years ago
tonycombocurve
commented
2 years ago See the Prefect migration guide for further details:
https://docs.prefect.io/migration-guide/?h=executor#conceptual-and-syntax-changes
jrbourbeau
commented
2 years ago Thanks for reporting @tonycombocurve. I believe updating this example is on @scharlottej13's radar
phobson
commented
2 years ago Tony, Sarah, and I chatted a bit in Slack. One confounding factor with even prefect's documentation is that it's example of connecting to a cluster with a TCP address and not a TSL address, for which distributed requires additional info (via ssl.SSLContext) to connect to. I opened an issue over at prefect_dask here: https://github.com/PrefectHQ/prefect-dask/issues/23
We also tried connecting the DaskTaskRunner to the coiled cluster by name, but here was no luck there:
import time
from prefect import task, flow
from coiled import Cluster
from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner
cluster = Cluster(name="quick-test", package_sync=True)
runner = DaskTaskRunner(
cluster_class="coiled.Cluster",
cluster_kwargs={
"name": cluster.name}
)
@task(name="shouter")
def shout(number):
time.sleep(0.5)
print(f"#{number}")
@flow(task_runner=runner)
def count_to(highest_number):
for number in range(highest_number):
shout.submit(number)
if __name__ == "__main__":
count_to(10)
ntabris
commented
2 years ago cluster = Cluster(...)
runner = DaskTaskRunner(
address=cluster.scheduler_address,
client_kwargs={'security': cluster.security}
)Ideally DaskTaskRunner would have API more like Client, as I suggested in https://github.com/PrefectHQ/prefect-dask/issues/23#issuecomment-1241207527
scharlottej13
commented
2 years ago @tonycombocurve wanted to let you know we now have a Prefect 2 Coiled docs page! As discussed in our call the other week, I think the section on using Prefect and Coiled with Dask collections is probably what you're looking for.
You can navigate to the Prefect 2 page from our Prefect 1 example. As more folks start transitioning to Prefect 2, we'll point people directly to Prefect 2 instead.
Thanks!
Your page on using Prefect for Workflow automation is out of date:
https://docs.coiled.io/user_guide/examples/prefect.html
The code examples are based on Prefect 1.0, which is no longer the current release. In particular, they have changed the API for invoking flows. In particular, there is no longer a run() method on the flow class. Instead, the flow is callable so you invoke it like this: flow(). They are also encouraging the use of the @flow on functions that implement the flow logic. Please consider updating your docs accordingly.
Thanks and keep up the great work!