PrefectHQ / prefect-ray

Prefect integrations with Ray
https://prefecthq.github.io/prefect-ray/
Apache License 2.0
63 stars 5 forks source link

Sometimes log output is missing #25

Closed ahuang11 closed 2 years ago

ahuang11 commented 2 years ago
from prefect import flow, task
from prefect_ray.task_runners import RayTaskRunner

@task
def count(number):
    with open(f"{number}.txt", "w") as f:
        f.write(str(number))
    print(number)

@flow(task_runner=RayTaskRunner())
def count_to(highest_number):
    for number in range(1, highest_number + 1):
        count.with_options(name=f"{number}").submit(number)

if __name__ == "__main__":
    count_to(3)

Below, it doesn't state 2-464db843-1 is completed; only 1 and 3.

13:34:57.620 | INFO    | prefect.engine - Created flow run 'adept-jackrabbit' for flow 'count-to'
13:34:57.620 | INFO    | Flow run 'adept-jackrabbit' - Using task runner 'RayTaskRunner'
13:34:57.620 | INFO    | prefect.task_runner.ray - Creating a local Ray instance
13:35:02.659 | INFO    | prefect.task_runner.ray - Using Ray cluster with 1 nodes.
13:35:02.671 | WARNING | Flow run 'adept-jackrabbit' - No default storage is configured on the server. Results from this flow run will be stored in a temporary directory in its runtime environment.
13:35:02.712 | INFO    | Flow run 'adept-jackrabbit' - Created task run '1-464db843-0' for task '1'
13:35:02.789 | INFO    | Flow run 'adept-jackrabbit' - Created task run '2-464db843-1' for task '2'
13:35:02.903 | INFO    | Flow run 'adept-jackrabbit' - Created task run '3-464db843-2' for task '3'
(begin_task_run pid=97109) 1
(begin_task_run pid=97108) 3
(begin_task_run pid=97109) 13:35:09.111 | INFO    | Task run '1-464db843-0' - Finished in state Completed()
(begin_task_run pid=97108) 13:35:09.097 | INFO    | Task run '3-464db843-2' - Finished in state Completed()
(begin_task_run pid=97107) 2
13:35:12.537 | INFO    | Flow run 'adept-jackrabbit' - Finished in state Completed('All states completed.')
ahuang11 commented 2 years ago

Cannot reproduce this anymore, closing.