search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
240 stars 46 forks source link

RasterJoin df.toLayer etc fail on non-standard CRS / Extent col names #503

Open vpipkt opened 3 years ago

vpipkt commented 3 years ago

This code:

val df = spark.read.raster //....
          .select(rf_tile($"red"), rf_extent($"red") as "red_extent", rf_crs($"red") as "red_crs")
          .toLayer(tlm)

Fails with error about column crs as crs not found. Pinpointed this to https://github.com/locationtech/rasterframes/blob/develop/core/src/main/scala/org/locationtech/rasterframes/extensions/RasterJoin.scala#L44

I believe in this case since there is no projected raster tile column (note use of rf_tile), it is trying to select crs and extent by name.

Should look for columns by type instead.

vpipkt commented 3 years ago

A highlight from the stack trace:

    at org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2225)
    at org.locationtech.rasterframes.extensions.RasterJoin$$anonfun$usePRT$1$3.apply(RasterJoin.scala:46)
    at org.locationtech.rasterframes.extensions.RasterJoin$$anonfun$usePRT$1$3.apply(RasterJoin.scala:45)
    at scala.Option.map(Option.scala:146)
    at org.locationtech.rasterframes.extensions.RasterJoin$.usePRT$1(RasterJoin.scala:45)
    at org.locationtech.rasterframes.extensions.RasterJoin$.apply(RasterJoin.scala:52)
    at org.locationtech.rasterframes.extensions.ReprojectToLayer$.apply(ReprojectToLayer.scala:52)
    at org.locationtech.rasterframes.extensions.DataFrameMethods$class.toLayer(DataFrameMethods.scala:215)