Storing key value pairs in redis without a table or a column names , and I am trying to build a df out of my data using the following code:
df = spark.read.format("org.apache.spark.sql.redis") \ .option("keys.pattern", pattern)\ .option("infer.schema",true)\ .load()
the code at this df level throwing this error :
.option("infer.schema",true)\ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 184, in load File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 134, in deco File "<string>", line 3, in raise_from pyspark.sql.utils.IllegalArgumentException: For input string: "<function true at 0x7fd215598830>"
any idea whats the issue ?
Storing key value pairs in redis without a table or a column names , and I am trying to build a df out of my data using the following code:
df = spark.read.format("org.apache.spark.sql.redis") \ .option("keys.pattern", pattern)\ .option("infer.schema",true)\ .load()the code at this df level throwing this error :.option("infer.schema",true)\ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 184, in load File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 134, in deco File "<string>", line 3, in raise_from pyspark.sql.utils.IllegalArgumentException: For input string: "<function true at 0x7fd215598830>"any idea whats the issue ?