search

daskos / epos

Write DAG Workflows like epic poems via Mesos, Marathon, Chronos, Spark, Dask
4 stars 1 forks source link

Cache chunked iterators #odo #14

Open nz87 opened 7 years ago

nz87 commented 7 years ago

Chunk size can exceed memory limit, to address this we should allow caching partial chunks to disk.