Upload Byte Array or non UTF-8 encoding strings to Key Value pairs

[tbreamgh]

Hi,

I have an application that writes from spark, but reads the content from python. I am able to control the encoding of the strings while reading/writing the content in python. However, I cannot see a way to either upload the raw byte array or a string in a different encoding that does not change the format.

I am uploading content in ISO_8859_1 and the UTF-8 encoding changes the string representation. Is there a work-around or a way to change the encoding, or upload the byte array.

Or is there a way to add support similar #304 but for the toRedisKV function

[jeremysong]

We have similar use case. In our case, we only need to upload RDD[(Array[Byte], Array[Byte])] to Redis.

One solution is to add a new method in redisFunctions:

  def toRedisByteKV(kvs: RDD[(Array[Byte], Array[Byte])], ttl: Int = 0)
                   (implicit
                    redisConfig: RedisConfig = RedisConfig.fromSparkConf(sc.getConf),
                    readWriteConfig: ReadWriteConfig = ReadWriteConfig.fromSparkConf(sc.getConf)) {
    kvs.foreachPartition(partition => setByteKVs(partition, ttl, redisConfig, readWriteConfig))
  }