Open raokrutarth opened 2 years ago
Hi @raokrutarth , can you try to configure in the following way?
spark.read
.format("org.apache.spark.sql.redis")
.option("table", key)
.option("infer.schema", true)
.option("host", AppConfig.settings.redis.host)
.option("port", AppConfig.settings.redis.port)
.option("auth", AppConfig.settings.redis.password)
.option("ssl", AppConfig.settings.redis.useTls)
.option("timeout", 5000)
.load()
That worked. Thanks. Any reason setting using spark.conf.set
does not work?
Reopening issue. Specifying creds for every read/write is very slow. Would be ideal to set the auth and establish a connection with redis once.
spark-redis takes the configuration options from org.apache.spark.SparkConf
. When you use sparkSession.conf.set("...", "...")
it sets the option in org.apache.spark.sql.RuntimeConfig
and those options are not propagated to SparkConf
.
In order to configure options in SparkConf
you have to pass them during the creation of SparkSession
, e.g.
val spark = SparkSession
.builder()
.master("local")
.config("spark.redis.port", "6379")
.getOrCreate()
We need to investigate if we should respect/support the options from RuntimeConfig
.
I have verified the redis instance is up on aws:
Versions:
Full exception from scala application:
Code snippets to reproduce: