search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
240 stars 46 forks source link

`spatial_index_partitions` on large (~2000) catalogs causes Kryo serialization issues. #502

Open metasim opened 3 years ago

metasim commented 3 years ago

Only 70% sure this is a bug, but need to be wary of it.

Same issue as this:

https://stackoverflow.com/questions/37709770/resolving-kryo-serialization-failed-buffer-overflow-spark-exception

Appears that adding the spatial index may get evaluated in a non-distributed way.