Impact of Redis licence change on KeyDB

[dceravigupta]

Just saw that Redis is moving to different licence starting version 7.4 and wondering what would be the impact of it on future releases of KeyDB?

https://redis.com/blog/redis-adopts-dual-source-available-licensing/

[JohnSully]

This project will remain with its original licensing.

KeyDB is currently based off Redis 6 primarily because of a lack of features that we needed from 7. However if we have enough time I do want to sync to the last true open source release of Redis.

[JohnSully]

As a followup there is renewed interest in the project given the license change.

KeyDB is used heavily at Snap serving >100 million QPS workloads. However we use the project primarily for set/get in cluster mode so most of my work is focussed on this. We are doing new work in FLASH and getting that to production readiness, that work is ongoing in the async_flash branch. A new release will be posted when that work is finished.

We have obviously fallen behind in dealing with the issue count here and would welcome community support in tackling this if anyone is interested in contributing.

[vwbusguy]

However if we have enough time I do want to sync to the last true open source release of Redis.

This is definitely a concern as Fedora ships redis-7, so the migration from redis to KeyDB would be a downgrade.

This development list discussion for Fedora is relevant: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/XVFFKU2NYB2Q3BQUYNANSDNE4VCJQ6KF/

[JohnSully]

@vwbusguy The Fedora community should understand that Redis labs was a unicorn company dedicated solely to Redis. While Snap is quite large as well they are obviously primarily focussed on the snapchat app not KeyDB. If the community wants to replace packages of Redis with KeyDB they need to understand this.

When I first joined Snap I thought I was going to be able to handle everything the same as before which was naive. It led to broken promises, so this time around I want to be very upfront with the community. As mentioned here and elsewhere I do welcome more maintainers if anyone else is willing to give their time to help support this.

[vwbusguy]

When I first joined Snap I thought I was going to be able to handle everything the same as before which was naive. It led to broken promises, so this time around I want to be very upfront with the community. As mentioned here and elsewhere I do welcome more maintainers if anyone else is willing to give their time to help support this.

Absolutely. I want to be clear that I'm also not speaking on behalf of the entire Fedora community here. That said, I appreciate your candor here and it's very relatable. I'm certainly happy to ask around Fedora about pitching in on this. I'm sure there are other OSS groups and Linux distros that might be willing to engage as well.

To that end, there's ongoing work right now to get KeyDB into Fedora and EPEL: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2270592

[dceravigupta]

Seems like Amazon is going down the path of forking Redis and a placeholder repo is already in place.

Another Redis fork created by one of the core member of Redis.

https://twitter.com/reconditerose/status/1771037984001032224?t=OE3tXhO0TfXE3NwvAJbpJA&s=19

Repo: https://github.com/madolson/placeholderkv

[vwbusguy]

It's not as much a problem of having a v7 upstream to rebase to, as I understand it, but rather the actual rebasing work that needs to be done. As in, someone brave/willing enough to rebase it on a BSD licensed v7 source and submit a Pull Request for it.

[JohnSully]

I am in discussions to potentially add maintainers and better align the code base to Redis 7.

If this bears fruit I will post updates here.

[Hisoka-J]

I am in discussions to potentially add maintainers and better align the code base to Redis 7.

If this bears fruit I will post updates here.

I'm concerned about whether KeyDB will continue to maintain compatibility with Redis 7.4 and later versions at the API level, such as user commands and management commands. I believe this is a crucial issue for me to continue using KeyDB. Because if there's incompatibility at the API level, it would increase the development complexity for our R&D personnel.

[JohnSully]

We definitely won't have direct source level compatibility anymore, since the upstream code is license incompatible.

This is a good thing because we had to make a ton of contortions to keep the code mergeable. Even then merges were bug prone and painful.

I can't promise we will reimplement every Redis proprietary feature but we will ensure we don't wantonly break compatibility. Most people use pre existing Redis clients and we would expect to maintain compatibility with those clients.

[vwbusguy]

This is a pretty common story right now - forgejo breaking with gitea, opensearch (formerly opendistro) with elastic, etc. Never underestimate the power of the open source community when driven by spite from a company that's done them wrong :smile:. If we want KeyDB to continue to work with Nextcloud, web frameworks, chatbots, etc., we'll make it work. I am grateful for the work done here in KeyDB am looking forward to better things to come from it than redis 7.4+ cross-compatibility, in ways that both benefit SnapChat and the rest of the world. Thanks, John et al!

[5455230]

We definitely won't have direct source level compatibility anymore, since the upstream code is license incompatible.

This is a good thing because we had to make a ton of contortions to keep the code mergeable. Even then merges were bug prone and painful.

I can't promise we will reimplement every Redis proprietary feature but we will ensure we don't wantonly break compatibility. Most people use pre existing Redis clients and we would expect to maintain compatibility with those clients.

历史的包袱不应该过于沉重,走出自己的路很重要.

[kurtisane]

From my point of view the main player is going to be Valkey since its backed by the Linux Foundation.

Yes its quite early on but seems to have a lot of support.

Would be awesome if KeyDb could be synched to that point (7.2.4) and further development and compatibility would be arranged together.

How far Redis itself differs from this point on is not anything of a concern to the open source community I guess.

[vwbusguy]

Open Source is not a zero sum game. There is plenty of room for both KeyDB and Valkey to exist and thrive, and since the licenses are compatible, there may be opportunities for collaboration between the projects. KeyDB offers some distinct feature set focuses from Valkey as well.

[caoyang1024]

The sharded pub/sub feature in Redis 7 looks promising. May I ask what the equivalent feature in KeyDB is, especially for horizontal scaling? Thanks

[rnz]

@JohnSully hi! Can you clarify whether KeyDB is still being developed or has development been discontinued?

[JohnSully]

It is still developed although it does much of what we need it to do (with the big exception of FLASH which should get a large update soon). I understand its not right for others and Valkey is a good option for those people.

On Mon, Jul 8, 2024 at 9:36 AM RNZ @.***> wrote:

@JohnSully https://github.com/JohnSully hi! Can you clarify whether cadeb is still being developed or has development been discontinued?

— Reply to this email directly, view it on GitHub https://github.com/Snapchat/KeyDB/issues/798#issuecomment-2214097059, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5W4AUQ6ADSEFYW746HQMLZLKIUFAVCNFSM6AAAAABFATAQCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUGA4TOMBVHE . You are receiving this because you were mentioned.Message ID: @.***>

[williamstein]

It is still developed although it does much of what we need it to do

Is there any chance you could enumerate which unique features from KeyDB you actually need, i.e., what the priorities are regarding maintenance of KeyDB going forward?

• multimaster replication?
• scaling via multithreading?
• FLASH (check - thanks for emphasizing the importance)

I'm using KeyDB in production, and actually appreciate that it is stable and works well, but I'm curious which features are high priority and which were just interesting things along the way, but aren't really needed for your main user who is paying the bills. Thanks for any hints.

[JohnSully]

The big thing it does is vertical scaling. You start to have issues around 500 cluster nodes and we have many clusters at that limit (and a bunch we’ve had to split up). In aggregate we have close to 10k nodes across all clusters running KeyDB.

FLASH will be a game changer but its blocked on not having async writes. This is currently in active development although we arent super responsive merging those. Its more of a we’ll merge it when its done type deal.

Subkey expires get used quite a bit.

We don’t use multi master and probably won’t do more work there.

-John

On Mon, Jul 8, 2024 at 10:09 AM William Stein @.***> wrote:

It is still developed although it does much of what we need it to do

Is there any chance you could enumerate which unique features from KeyDB you actually need, i.e., what the priorities are regarding maintenance of KeyDB going forward?

• multimaster replication?
• scaling via multithreading?
• FLASH (check - thanks for emphasizing the importance)

I'm using KeyDB in production, and actually appreciate that it is stable and works well, but I'm curious which features are high priority and which were just interesting things along the way, but aren't really needed for your main user who is paying the bills. Thanks for any hints.

— Reply to this email directly, view it on GitHub https://github.com/Snapchat/KeyDB/issues/798#issuecomment-2214180821, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5W4AXOW6LQ2Y2TX6KWPALZLKMQBAVCNFSM6AAAAABFATAQCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUGE4DAOBSGE . You are receiving this because you were mentioned.Message ID: @.***>

[JohnSully]

The other thing I want to do is have a global expire since a lot of our workloads use the same expire for all keys.

But this is still in the idea phase.

-John

On Mon, Jul 8, 2024 at 10:13 AM John Sully @.***> wrote:

The big thing it does is vertical scaling. You start to have issues around 500 cluster nodes and we have many clusters at that limit (and a bunch we’ve had to split up). In aggregate we have close to 10k nodes across all clusters running KeyDB.

FLASH will be a game changer but its blocked on not having async writes. This is currently in active development although we arent super responsive merging those. Its more of a we’ll merge it when its done type deal.

Subkey expires get used quite a bit.

We don’t use multi master and probably won’t do more work there.

-John

On Mon, Jul 8, 2024 at 10:09 AM William Stein @.***> wrote:

It is still developed although it does much of what we need it to do

Is there any chance you could enumerate which unique features from KeyDB you actually need, i.e., what the priorities are regarding maintenance of KeyDB going forward?

• multimaster replication?
• scaling via multithreading?
• FLASH (check - thanks for emphasizing the importance)

I'm using KeyDB in production, and actually appreciate that it is stable and works well, but I'm curious which features are high priority and which were just interesting things along the way, but aren't really needed for your main user who is paying the bills. Thanks for any hints.

— Reply to this email directly, view it on GitHub https://github.com/Snapchat/KeyDB/issues/798#issuecomment-2214180821, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5W4AXOW6LQ2Y2TX6KWPALZLKMQBAVCNFSM6AAAAABFATAQCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUGE4DAOBSGE . You are receiving this because you were mentioned.Message ID: @.***>

[rnz]

@JohnSully Thank you for your responses! Unfortunately, I often observe conditions similar to deadlock on 6.3.4, and under a fairly light load:

How often do 6.3.4 nodes have deadlocks in your clusters? Do you have statistics?

[fxck]

@rnz are those similar to https://github.com/Snapchat/KeyDB/issues/830? We are also encountering those, under light loads.