Open justinTM opened 2 years ago
Hi @justinTM , could you try it with spark-redis 3.1.0? Also, could you try to specify connection parameters when creating Spark session rather than as dataframe save options?
I am also experiencing the same issue.
When I do
import redis
r = redis.StrictRedis(host=redis_host, port=int(redis_port), ssl=True, decode_responses=True)
print(r.ping())
I can see True
being printed.
However, I got Caused by: java.net.ConnectException: Connection refused (Connection refused)
when I do
loadedDf = (
spark
.read
.format("org.apache.spark.sql.redis")
.option("spark.redis.host", redis_host)
.option("spark.redis.port", redis_port)
.option("spark.redis.ssl", "true")
.load()
)
loadedDf.show()
I also try
from pyspark.sql import SparkSession
another_spark = (
SparkSession
.builder
.config("spark.redis.host", redis_host)
.config("spark.redis.port", redis_port)
.config("spark.redis.ssl", "true")
.getOrCreate()
)
loadedDf = (
another_spark
.read
.format("org.apache.spark.sql.redis")
.load()
)
loadedDf.show()
However, the error are still the same
P.S. I am using 3.1.0
I can connect to the server with the following code
loadedDf = (
spark
.read
.format("org.apache.spark.sql.redis")
.option("host", redis_host)
.option("port", redis_port)
.option("ssl", "true")
.option("infer.schema", True)
.load()
)
loadedDf.show()
Same Exception using master branch of spark-redis and spark 3.3.0
sql( "create table if not exists leads( id int, first_name string, last_name string, email string, gender string, city string, country string) using org.apache.spark.sql.redis options (table 'leads')" )
I can't connect to an AWS MemoryDB cluster using SSL; I get an exception:
redis.clients.jedis.exceptions.JedisConnectionException
.Using the TablePlus app on macOS, I was able to connect with the same params as in DataBricks (using this library):
But when executing on DataBricks cluster:
Any ideas?