search

stackabletech / spark-k8s-operator

Operator for Apache Spark-on-Kubernetes for Stackable Data Platform
https://stackable.tech
Other
51 stars 2 forks source link

Misc #254

Closed razvan closed 1 year ago

razvan commented 1 year ago

When using k8s mem reqs of 512Mi:

 Exception in thread "main" java.lang.IllegalArgumentException: System memory 129761280 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.   │
│     at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:221)                                                                                                                              │
│     at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:201)                                                                                                                                     │
│     at org.apache.spark.SparkEnv$.create(SparkEnv.scala:325)                                                                                                                                                                   │
│     at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:196)                                                                                                                                                          │
│     at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)                                                                                                                                                    │
│     at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)                                                                                                                                                            │
│     at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2740)                                                                                                                                                     │
│     at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1026)                                                                                                                               │
│     at scala.Option.getOrElse(Option.scala:189)                                                                                                                                                                                │
│     at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1020)                                                                                                                                          │
│     at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)                                                                                                                                                               │
│     at org.apache.spark.examples.SparkPi.main(SparkPi.scala)                                                                                                                                                                   │
│     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                          │
│     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                        │
│     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                │
│     at java.base/java.lang.reflect.Method.invoke(Method.java:566)                                                                                                                                                              │
│     at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)                                                                                                                                            │
│     at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)                                                                                                                │
│     at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)                                                                                                                                                  │
│     at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)                                                                                                                                                       │
│     at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)                                                                                                                                                      │
│     at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)                                                                                                                                            │
│     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)                                                                                                                                                       │
│     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)