search

RedisLabs / spark-redis

A connector for Spark that allows reading and writing to/from Redis cluster
BSD 3-Clause "New" or "Revised" License
940 stars 372 forks source link

Will this repo be maintained anymore? #393

Open MrPowers opened 2 months ago

MrPowers commented 2 months ago

First off, thank you for building this important library for the Spark Community!

I see that the last commit for this repo was two years ago and there are open issues to add support for Spark 3.4 and Spark 3.5.

Is there a project maintainer that would be willing to merge PRs and make releases for new Spark versions?

banker commented 2 months ago

@MrPowers Of course! And yes, we'd be happy to look over PRs and release a new version. What's your biggest need right now?

MrPowers commented 2 months ago

Thanks @banker! Would you be willing to push up artifacts that are Spark 3.4 & Spark 3.5 compatible? Would you be willing to refactor to make spark-redis Spark Connect compatible?

mpasa commented 2 months ago

Also, as Spark 4.0 is dropping support for Scala 2.12, it would be great to see it cross-compiled for both 2.12 and 2.13. This is currently blocking our version upgrade and we'd need to drop the library if not possible.

newfront commented 1 month ago

I added a branch for 4.0 (https://github.com/RedisLabs/spark-redis/pull/394). @banker there is one open question regarding expected behavior for the RedisNode.idx index. With the modifications to Jedis they deprecated conn.clusterSlots in favor of conn.clusterShards which provides different behavior. All cluster slots per Node are returned - so this might change the way things work. If you have a chance to look things over I'd be more than happy to change the logic.