Closed JeroenVerstraelen closed 1 year ago
Issue #183 could be related.
the proposed fix is implemented, which reduces the number of jobs on tsservice a lot. I do believe number of partitions is now higher, and partitioning is not optimal with lots of empty tasks. We can probably derive an sparse partitioner based on dates gathered from catalog, without requiring a spark job.
done
When looking in the spark UI we can see that even for very small RDDs the stage related to createPartitioner can sometimes take up to 100 seconds (20 seconds in garbage collection). We should add a check to see if there are for example less than 10 keys to be processed. If that's the case, we can automatically use the SpaceTimeByMonthPartitioner.