search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
455 stars 34 forks source link

Could not parse Master URL: 'yarn' #197

Closed zenwan closed 1 year ago

zenwan commented 1 year ago

image

How to solve this problem？ if change yarn to local, will work well

Jolanrensen commented 1 year ago

If you use %use spark, then all configuration for spark should be put in the %use magic. As described here: https://github.com/Kotlin/kotlin-spark-api/wiki/Jupyter This would turn your first statement into

%use spark(spark=3.2.0, scala=2.13, v=1.2.3, spark.home="/opt/app/spark-3.2.0", spark.config.file="/opt/app/spark-3.2.0/conf/spark-defaults.conf", spark.app.name=kotlin_spark_demo, spark.master=yarn)

Aside from that, the %use spark magic at the moment does not include the spark-yarn dependency by default. (I made an issue for that) This means you'll have to include it manually in another cell before the %use spark statement like

USE {
    dependencies("org.apache.spark:spark-yarn_2.13:3.2.0")
}

Let me know how that works for you :)