locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
246 stars 45 forks source link

Fix for SparkML memory pressure issues #344

Closed metasim closed 5 years ago

metasim commented 5 years ago

Fixed SparkML memory pressure issue caused by unnecessary reevaluation, overallocation, and primitive boxing.

Closes #343

metasim commented 5 years ago

@vpipkt I'm really sorry but I accidentally did a clean and killed off the ExplodeTiles stats. They weren't hugely different... < 10%? But the improvement may have other benefits depending the spark.sql.tungsten.enabled setting.