merge of system properties,application.conf @ jar:file:/.../rasterframes-uberjar_2.11-0.9.0.jar!/application.conf: 1: No configuration setting found for key 'rasterframes'

[jornfranke]

Rasterframes: 0.9.0 Spark: 2.4.0 Python

I tried for our cluster to load the jar of rasterframes from a specific location as follows:

import pyrasterframes
from pyspark.sql import SparkSession

import timeit
from datetime import timedelta
spark = SparkSession\
    .builder\
    .config("spark.driver.extraClassPath", "hdfs://location/rasterframes-uberjar_2.11-0.9.0.jar")\
    .config("spark.jars", "hdfs://location/rasterframes-uberjar_2.11-0.9.0.jar")\
    .config('spark.executor.memory', "2g")\
    .config("spark.driver.memory", "3g")\
    .config("spark.kryoserializer.buffer.max", "1g")\
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.kryo.registrator", "org.locationtech.rasterframes.util.RFKryoRegistrator") \
    .getOrCreate()
spark=spark.withRasterFrames()  

from pyrasterframes.rasterfunctions import *
from pyspark.sql.functions import lit
df = spark.read.format('raster').load("hdfs://...../U2018_CLC2018_V2020_20u1.tif")

and received the following error

Caused by: com.typesafe.config.ConfigException$Missing: merge of system properties,application.conf @ jar:file:/tmp/spark-35a6a356-2b67-49d0-9bde-1ec91d372b6d/rasterframes-uberjar_2.11-0.9.0.jar!/application.conf: 1: No configuration setting found for key 'rasterframes'
    at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
    at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
    at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
    at org.locationtech.rasterframes.package$.rfConfig(package.scala:50)
    at org.locationtech.rasterframes.package$.<init>(package.scala:54)
    at org.locationtech.rasterframes.package$.<clinit>(package.scala)

I checked the .jar. It contains the application.conf and it does not contain the key. However, there is a file called reference.conf, which contains it. I checked also the original rasterframe and there it is the same. The uberjar I created contains basically all 4 rasterframe libraries (cf. https://mvnrepository.com/artifact/org.locationtech.rasterframes) of course properly merged using a buildfile etc.

[metasim]

Creating your own "uberjar" from the artifacts on mvnrepository.com isn't going to have all the transitive dependencies included in it. My recommendation is to get the pyrasterframes pip package and grab the jar assembly from it. Or you can check out the source and run sbt pyrasterframes/assembly.

[jornfranke]

I managed to merge it properly using another merging strategy.