Sparkling Water not properly configuring RAM on Databricks

[bondgy]

Providing us with the observed and expected behavior definitely helps. Giving us with the following information definitively helps:

• Sparkling Water/PySparkling/RSparkling version: Sparkling Water is 3.40.0.4 (3.3 JAR), PySparkling is 3.40.0.4-1 (3.3), RSparkling is 3.40.0.4 (3.3), H2O is 3.40.0.4
• Hadoop Version & Distribution: Databricks runtime 12.2 LTS, Apache Spark 3.3.2, Scala 2.12
• Execution mode YARN-client, YARN-cluster, standalone, local: Databricks starting internal sparkling water clusters
• YARN logs in case of running on yarn. To collect such a logs you may run yarn logs -applicationId <application ID> where the application ID is displayed when Sparkling Water is started: No explicit warnings/errors
• H2O & Spark logs if not running on YARN. You can find these logs in Spark work directory: No explicit warnings/errors
• Are you using Windows/Linux/MAC?: Databricks
• Spark & Sparkling Water configuration including the memory configuration: Databricks runtime 12.2 LTS, Apache Spark 3.3.2, Scala 2.12, driver has 512GB memory and 64 cores and 6 non-scaling workers with 384 GB memory and 96 cores (64GB memory and 16 cores each). Runtime is 12.2.x-scala2.12

Please also provide us with the full and minimal reproducible code. All of these outputs below refer to a just-started Databricks cluster with no other notebooks/jobs running. When calling

H2OContext.getOrCreate()

The total cluster memory is shown in the output of the cell box as ~60GB of memory for both PySparkling and RSparkling clusters. The memory is accessible when using a normal H2O cluster. When running

h2o.init(max_mem_size="200g")

a cluster with ~200GB of memory is created. Since Databricks doesn't seem to explicitly set the spark.driver.memory and spark.executor.memory properties, I tried setting these to varying levels, but it did not change the cluster size. Since the cluster size is too small for my modeling, when running more complex tasks (e.g., GridSearch or XGBoost with a large amount of trees or depth) the cluster itself becomes unreachable. Is there a hidden property that must be set explicitly to fully utilize Databrick's RAM or is there a bug?

[bondgy]

I see this is actually a limitation of Databricks now. The AWS node type limits the amount of RAM a cluster can have for its executors, and I don't see a way to get around this except by purchasing types that allow more, which isn't super ideal.