When echodataflow is set up to use an external dask cluster, the memory consumption of this cluster seems to continuously grow (beyond the size of the files that are currently being processed): i.e. there seems to be unsuccessful garbage collection.
Need to investigate more if that can be avoided. @Sohambutala share what you have tried.
When echodataflow is set up to use an external dask cluster, the memory consumption of this cluster seems to continuously grow (beyond the size of the files that are currently being processed): i.e. there seems to be unsuccessful garbage collection.
Need to investigate more if that can be avoided. @Sohambutala share what you have tried.