Does spark-redis support hash value compression on read/write ?

RedisLabs / spark-redis

A connector for Spark that allows reading and writing to/from Redis cluster

BSD 3-Clause "New" or "Revised" License

935 stars 368 forks source link

Does spark-redis support hash value compression on read/write ? #296

Open gwwallace opened 3 years ago

gwwallace commented 3 years ago

Is there an option to specify that values in a Hash are compressed ?

fe2s commented 3 years ago

Hi @gwwallace , there is no such option

jonathansp commented 2 years ago

Is there any alternative @fe2s ?

rfan-debug commented 2 years ago

We're loading pretty large values to hash as well. It would be great to support compression!

====

As an alternative, we compress the value in the dataframe right before writing. It is a bit cumbersome but still manage-able. (And I am not quite sure if adding compression as a spark.redis.compression option is over-engineering of this library.