Closed jcus0006 closed 6 months ago
https://www.anyscale.com/blog/analyzing-memory-management-and-performance-in-dask-on-ray
Dask on Ray uses a shared memory object store in the scheduler, the absence of which likely causes excessive overhead in Dask.
In this regard, Ray or Dask on Ray, may be considered, shortly.
This was eventually not opted for, but Dask Strategy 2 replicates the same idea
https://www.anyscale.com/blog/analyzing-memory-management-and-performance-in-dask-on-ray
Dask on Ray uses a shared memory object store in the scheduler, the absence of which likely causes excessive overhead in Dask.
In this regard, Ray or Dask on Ray, may be considered, shortly.