jacobtomlinson
commented
4 years ago I would be happy to explore this but it is worth noting that Dask manages it's own memory swapping and doesn't use the built in system swap. Futures are moved in and out of memory into the temporary worker space on disk.
If you are having trouble with workers running out of memory this may be related to your dataframe partitioning.
It would be nice to specify swap memory, especially when dealing with very large DataFrames
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definitions (search for maxSwap)