Memory limits should be measured in bytes, not # of partitions

go-sif / sif

Sif is a framework for fast, predictable, general-purpose distributed computing in the map/reduce paradigm.

Apache License 2.0

32 stars 3 forks source link

Memory limits should be measured in bytes, not # of partitions #17

Closed Ghnuberath closed 4 years ago

Ghnuberath commented 4 years ago

Partition size will vary between stages, so measuring cache size in # of partitions leads to an inefficient use of memory for stages with smaller partitions.