Open nohanaga opened 1 year ago
Hello!
I think That's due to the codec in the RFRasterSource companion object; which implementation lives here: https://github.com/locationtech/rasterframes/blob/develop/core/src/main/scala/org/locationtech/rasterframes/encoders/StandardEncoders.scala#L87
I would assume that's due to the frameless dependency on shapeless.
Unfortunately at this point the sbt assembly should be properly built and shaded to avoid shapeless / frameless deps issues.
That's a good question when DB plans to upgrate their environment to depend on the more up to date lib versions.
@pomadchin Thanks for the response. Is this problem due to the version of the library on the Databricks side? Should I look into the shapeless library?
I think more into the way rasterframes is loaded; I feel like there can be an issue with how RF jar is built / used within the cluster.
Could you trigger somehow a more precise error i.e. by calling https://github.com/locationtech/rasterframes/blob/develop/core/src/main/scala/org/locationtech/rasterframes/encoders/StandardEncoders.scala#L87 explicitly from the SBT shell?
I kinda had a feeling that everything that needs to be shaded is shaded 🤔 https://github.com/locationtech/rasterframes/blob/cc0264438a4a42d4424343401e7001906c2ce6e6/project/RFAssemblyPlugin.scala#L51-L66
I used pyrasterframes v0.10.1 with Databricks 9.1 LTS ML (includes Apache Spark 3.1.2, Scala 2.12). I get the following error and cannot access the dataframe.
Code
Prerequisite
Basically, I used this explanation as a guide. https://www.databricks.com/notebooks/rasterframes-notebook.html
The above file was created and registered in the Cluster-scoped init script.
sbt publishLocal
to obtain pyrasterframes .jar and .whl. a. pyrasterframes-assembly-0.10.1.jar b. pyrasterframes-0.10.1-py3-none-any.whl Register these two files in the cluster library.Execution conditions
Confirmed on Spark Notebook.
gdal_version, build_info
Java Version
I would like to use pyrasterframes on Databricks, so please investigate.