Purge old Dask futures when no longer needed

[RobJBarr]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar request and didn't find it.
• [X] I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

In the current implementation of the DaskTaskRunner, the Dask futures for each task are maintained in a dictionary, even when no longer needed in downstream tasks. This results in a reference to the future being maintained on the client side so it is never cleaned up from the workers

Describe the proposed behavior

An option to purge Dask futures which are no longer needed is provided in the DaskTaskRunner. Perhaps this could take the form of a callback once a prefect future is completed to check if all its upstream tasks have any further downstream dependents, and if not then delete the futures so they can be garbage collected

Example Use

No response

Additional context

No response

[desertaxle]

Thanks for the issue @RobJBarr! We have a new implementation of the DaskTaskRunner that will released alongside our upcoming 3.0 release that no longer keeps references to Dask futures. I think that will resolve this issue that you're seeing. If you want to give it a try, you can install prefect==3.0.0rc1 and prefect-dask==0.3.0rc1.