[Discourse] How do I make dask memoize intermediate task results on a file system? - Githubissues

coiled / dask-community

Issue tracker for the Dask community team

MIT License

2 stars 0 forks source link

[Discourse] How do I make dask memoize intermediate task results on a file system? #1429

Open github-actions[bot] opened 1 year ago

github-actions[bot] commented 1 year ago

I have a data analysis pipeline with many functions that consume and produce DataFrames in a DAG-like way.

What I’m trying to accomplish

i want dask to save the output of any DataFrame-returning task (each delayed?) as a parquet file on a file…

Would you like to know more?

Read the full article on the following website:

https://dask.discourse.group/t/how-do-i-make-dask-memoize-intermediate-task-results-on-a-file-system/1138