Spark jobs with lots of partitions can crash if the driver is too small, for example with the error,
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 731259 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.
Set explicit guidelines for each job about cluster setup.
This will come once we have the continuous integration pipeline running on multitask jobs with a different setup for each use case (ingest vs etl vs regressions etc)
Spark jobs with lots of partitions can crash if the driver is too small, for example with the error,
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 731259 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.Set explicit guidelines for each job about cluster setup.
This will come once we have the continuous integration pipeline running on multitask jobs with a different setup for each use case (ingest vs etl vs regressions etc)