SparkR exposes the number of partitions in the API. Plyrmr would like to make that an implementation detail. Right now the problem is swept under the carpet by slapping a few constants here and there. First step is to replace constants with some setting, next is to find a way to decide a reasonable number of partitions behind the scenes.
SparkR exposes the number of partitions in the API. Plyrmr would like to make that an implementation detail. Right now the problem is swept under the carpet by slapping a few constants here and there. First step is to replace constants with some setting, next is to find a way to decide a reasonable number of partitions behind the scenes.