valkey-io / valkey

A flexible distributed key-value datastore that is optimized for caching and other realtime workloads.
https://valkey.io
Other
17.56k stars 663 forks source link

Merging efforts with Redict #18

Open ddevault opened 8 months ago

ddevault commented 8 months ago

Hi, I'm one of the people working on the Redict fork, and we would like to merge efforts if possible. Bringing the discussion here from #8 (and various private conversations) where there is some additional context.

The only thing we're really committed to is the name and the use of the LGPL license. If the folks working on this fork can get on board with that, then it does not make sense to split the community in two, but we are committed to the copyleft approach so we will need to get consensus on this to merge forks. The rationale behind LGPL has been explained in a few different places, but I've summarized it here.

As far as practical matters are concerned, there is the matter of infrastructure and communication. We're using Codeberg rather than GitHub right now, which should be very familiar to GitHub users and has the advantage of being free software, which we are strongly in support of using. I hope you're willing to give it a shot, but I don't think it's a hill worth dying on. Discord on the other hand, I can't speak for everyone involved in Redict but I think it's a very hard sell. We're using IRC right now, but we could be convinced to use Matrix, which is a bit more user friendly.

stalkerg commented 8 months ago

There is always politics.

yes, sure, but I hope you got my point what it's extra tension, and LGPL a little mitigate it. Anyway, thank you for your initiatives!

PS with BSD Redis lab can use a such code without any feedback, it's sounds unfair in the current situation.

CRConrad commented 8 months ago

The key difference here is there is no "Redis Inc" to change the license. This is a community run project, similar to other community projects that have maintained permissive languages the entire time

There is no such company yet. Someone could go found one tomorrow. At one point, there was no Redis, Inc, either, and then someone went and founded it.

mattsta commented 8 months ago

There is no such company yet. Someone could go found one tomorrow.

I think the reality is there's no profit in this anymore.

Software like redis isn't a core product. It's just a side car component complementing larger platforms. It's a fungible plastic fork in a world full of takeout and nobody wants to venture fund a plastic fork factory.

When redis started in 2009, the world was just waking up to the reality of "you can't use disk for user services because now networks are fast and disks are slow, so everything must be in memory." In 2009, having a server with 32 GB RAM was impressive — and today my watch has 32 GB RAM and modern SSDs with their direct-to-CPU interface lanes are almost faster than RAM from 2009 (related: linus released a fun video on optane yesterday too https://www.youtube.com/watch?v=Al93JD5GExY)

Today you can buy a single 1U server with hundreds of cores and 3 TB RAM for less than $100,000 USD. That's almost more capacity in one single 1.75" tall server than an entire 10 rack data center deployment from 15 years ago.

Here's an article from 2008 showing RAM was around $125/GB: https://www.tomshardware.com/reviews/ddr3-1333-speed-latency-shootout,1754-26.html. Today, RAM costs around $2/GB and 1 TB of high quality SSD is around $50/TB (unless you use Apple Pricing of $400/TiB).

Basically, my laptop today is more powerful than an entire 42U rack of servers 15 years ago. This also means people care less about high quality high performance software since even really poorly architected lazy programs just get fast with modern hardware (until you get hit with actual intractable scalability bottlenecks, but nobody thinks ahead that far anymore).

The benefit of simple key-value databases got an industry/VC "noSQL" hype cycle between 2010-2016, but once you fall out of the hype cycle the world moves on.

From what I've seen in the industry, it's mostly "nobody cares" anymore. I've been looking for a new job for a while and it appears there are only two roles left in the world:

Yesterday AWS gave anthropic almost $3 billion in funding/support, but when was the last time anybody seriously funded a database (and I mean seriously and not just a bridge round or a pity "we wasted money on you so far, so here's a little more" round)? Nobody funds databases anymore. We just live off talented passionate people throwing off free software worth billions of dollars in surplus value nobody can capture except hosting providers (and you can't "out hosting-provider" the top 3 platforms yourself).

I've always thought a high efficiency high performance legacy redis-like platform is a great fit for what people call IoT — which includes things like tens of thousands of modern 5G distributed base stations and edge deployments (or anything you call a "station" or "edge" or "CPE" as a remote deployment). In remote, non-data-center, hardware deployment situations you really care about minimizing hardware footprint/spend via memory+compute efficiency and reliability (thus not duplicating hardware unnecessarily on-site where you have to roll a truck to fix things). Getting your products used in forward-deployed hardware is more of an enterprise big S&S sales cycle though instead of the open source "blog post -> clever intern trying new things for resume-oriented-development -> new ideas deployed as internal services -> becomes production service -> suddenly you're trapped into needing CYA support contracts" pipeline.

overall? Best outcome: somebody pays "just for the good of the world" for experienced high productivity experts to maintain high quality public software under a guaranteed growth+maintenance agreement for decades. Modern world: eat it up and chew it out then complain when your cud doesn't taste like strawberries and rainbows anymore.

stalkerg commented 8 months ago

@madolson hmm what you decided after all? Will your collaborate with Redict or not?

PeterZaitsev commented 8 months ago

In my opinion license change compared to what Redis used to have is a poor choice, because it ads another layer of friction - if you are serious organization which cares about license compnatibility you will need to run analyses here. In particular LGPL is not that common license those days.

Also to successfully compete against Redis you want this software to be as usable as possible - easily embeddible in all kind of projects - both open source and proprietary.

"Preventing someone else to pull what Redis did" is achieved with proper Governance with Trademark ownership. Imagine if Linux Foundation (or soe other non profit) would have owned Redis trademark ? The only thing Redis could do is to create another project and make that project under SSPL license, as well as stop contributing to Open Source version

reconbot commented 7 months ago

I work at large companies that require license reviews, and for many previous employers I've assisted in those reviews. LGPL is not added friction. We already know about it, it's already on the list.

Personally I like it. GPL can be tough to reason about (or annoying to implement) but LGPL you don't have to care about if you're a user just like MIT. And then you get the sharing potential for modifications and improvements. I love that.

I do agree a foundation governance is the best defense we need and I'm glad this project has it.

PeterZaitsev commented 7 months ago

@reconbot My point is not about LGPL being bad license, but rather if the goal to provide drop-in replacement and seamless migration for users of old Open Source Redis versions, it is best achieved if license is the same.

While being Great license LGPL is not drop in for BSD in all cases.

yawaramin commented 7 months ago

Well, there are several goals, aren't there? And there may have to be some compromise between them, to ensure the best possible outcomes. Two goals just off the top of my head:

Obviously there may be there goals held by others. But LGPL seems like a pretty good compromise between the needs of the community and the health of the OSS project going forward. I don't think @madolson is strongly opposed to LGPL as long as the project is on GitHub, nor is @ddevault strongly opposed to GitHub as long as the project is licensed as LGPL. It seems like a compromise should be reachable?

PeterZaitsev commented 7 months ago

The thing is LGPL does not solve the problem, because it is not the license problem. Think about issue with RedHat CentOS cituation for example.

The issue is Governance, Trademark and means of distribution ownership. If Redis trademark would be owned by some foundation as in case of Valkey they would need to fork off to differently named project which does not have the same value.

If product is LGPL licensed though it does not prevent some "corpotate interests" for example to keep their improvements to only version which is available as a cloud, even GPL does not as you can see with AWS Aurora MySQL.

Furthermore you can look at the projects like Kubernetes or PostgreSQL both of those Permissively Licensed, have high commercial "hijacking" value but Found governed and you can see they have not been taken over by corporate interests

In the nutshell the license change to LGPL will offer very limited protection compared to foundation governance alone while create additional friction.

Conan-Kudo commented 7 months ago

The thing is LGPL does not solve the problem, because it is not the license problem. Think about issue with RedHat CentOS cituation for example.

Seriously? This is nothing like that. The sources remain available to everyone through CentOS Stream, and the RHEL SRPMs are still available to customers. Ultimately, nobody "lost" the rights afforded to them as Red Hat customers if they use RHEL.

Conan-Kudo commented 7 months ago

In the nutshell the license change to LGPL will offer very limited protection compared to foundation governance alone while create additional friction.

Actually, it creates a very important piece of friction: proprietary changes to the core code is no longer possible. And with distributed copyright, the license can never be changed again without asking everyone who contributed, guaranteeing that a license change isn't going to happen willy-nilly.

That's worth a lot for a lot of people, myself included.

yawaramin commented 7 months ago

If product is LGPL licensed though it does not prevent some "corpotate interests" for example to keep their improvements to only version which is available as a cloud

@PeterZaitsev but it solves the problem that the project owner was able to relicense the project because it had an extremely permissive license. In the future, Redict won't be vulnerable to this any more. But Valkey will (as things stand today).

Governance teams come and go. In the beginning everything is great and everyone gets along. Then people and teams change and corporate interests take over. Why leave it to chance? Even a slightly stronger copyleft license like LGPL prevents the issue at its root rather than relying on the goodwill of all parties for all time to come.

madolson commented 7 months ago

I appreciate everyone being so civil on this thread, I know it's a contentious topic and I'm glad everyone here has the opportunity to convey their thoughts.

I do agree a foundation governance is the best defense we need and I'm glad this project has it.

This was my main priority, and I'm really happy with the support we've gotten from the LF and I think the future (at least the couple of year outlook) seems pretty secure to me. I'm more interested in keeping Valkey community driven in the long term, as opposed to one where it remains open but still potentially controlled by corporations that don't benefit the community. The case that yawaramin mentioned about corporate interests taking over already feels like a failure state to me, since they would have needed to acquire a majority of seats on the technical steering committee, and that means the community would have not tried to stop them. Just because they can't relicense it, doesn't mean they couldn't still use LGPL license to advance their aims.

I suppose my question to advance this discussion is are people going to avoid contributing if we don't try to move to a more restrictive, but still FOSS, license?

yawaramin commented 7 months ago

...they would have needed to acquire a majority of seats on the technical steering committee, and that means the community would have not tried to stop them.

Sure, this seems unlikely now. But Redis is 15 years old. In another 5 or ten years from now, who knows? At the very least it seems pretty clear that Open Source software will continue to undergo churn. For example, who would have thought that, when antirez stepped down from Redis and the new maintainers promised it would stay OSS, that this would not be the case? Did the community get a chance to prevent relicensing? No, everyone was blindsided. It seems prudent to put in place a hard guarantee that despite any failure state, the Valkey project could not be co-opted in that way.

But of course, as you said, the crucial question is what contributors will do. And I guess we'll see that play out with Valkey and Redict.

PeterZaitsev commented 7 months ago

I will say more, in my opinion pre-venting ability to have derived software with different license is actually harmful, as it is companies which end up making money around the project are those who end up spending resources on it. What should be prevented is the Open Source Project changing it license as it happened with Redis, which is governance issue.

Look at PostgreSQL vs MySQL for example - one of the reasons PostgreSQL is so successful, many companies worldwide are free to build the products, including many commercial ones around PostgreSQL, where MySQL with GPL license prevented that (besides the cloud).

Also note very big part of use case is NOT really protected by LGPL license (or even choosing GPL) - Many Valkey project members are Cloud vendors who can offer cloud version with some proprietary improvements, and I am quite sure some will.

In the nutshell if the focus of the project is about attracting as many users and contributors as possible, keeping permission is the best, and I think this should be the main focus at this early point.

yawaramin commented 7 months ago

MySQL with GPL license prevented that

The obvious counterexample being Linux with GPL license being used in almost every device and workload you can think of.

if the focus of the project is about attracting as many users and contributors as possible, keeping permission is the best

And if there are multiple goals that can be reconciled together, then a stronger copyleft license is quite possible.

ddevault commented 7 months ago

I suppose my question to advance this discussion is are people going to avoid contributing if we don't try to move to a more restrictive, but still FOSS, license?

I think that most of what needs to be said on this issue has been said, but I do want to quickly correct this framing of the LGPL et al as "more restrictive". They are not more "restrictive" -- any license which restricts freedom of use is by definition non-free and/or non-open. They have some obligations, which is not the same thing. MIT also has obligations, like preserving the copyright notice. Permissive licenses like MIT come from the view where "freedom" means relatively few obligations; copyleft licenses like LGPL posit that freedom is a guarantee of rights. I don't think copyleft licenses are more "restrictive" but they are certainly more free.

mattsta commented 7 months ago

Permissive licenses like MIT come from the view where "freedom" means relatively few obligations; copyleft licenses like LGPL posit that freedom is a guarantee of rights. I don't think copyleft licenses are more "restrictive" but they are certainly more free.

The historical artifact around "free software" is amusing because all the "free/libre!" software ideas were made by people who grew up in literal hippie communes and had everything given to them by a shared community without needing to do the whole capitalism "profit every day or die" dance.

It's a great ideology if you have an unlimited source of personal funding and low-to-no personal growth ambitions (or live in low/no cost of living countries, or just get placed into a lifetime university appointment to promote "free software"), but we've seen it doesn't work to survive. I can release world class software, get it used by a million developers, have it deployed inside multiple multi-trillion-dollar companies to subsidize their private profit machines, yet the creators still can't afford a place to live.

Every prominent "open source" developer/architect/maintainer survives off social status (or temporary VC funding before they realize "open source" isn't a business model) and not their actual work output (see: guido hopping between 5 companies just to keep getting paid because nobody is ever going to really pay for python, etc).

We lost the game of "unlimited user freedom, but restricting private rights" a while ago and I'm not sure we can ever fully get it back. If you add restraints, everybody will duplicate your project to work around you anyway. If you don't add restraints, everybody will copy your project directly and never compensate developers for global value created. 🤷 — so the game becomes "do we do good work and enforce restrictions, but then become obsoleted by funded copies" or "do we do good work, let the world copy it with no restrictions, then maintain some social status as 'creator of the thing everybody uses but nobody pays for?'"

madolson commented 7 months ago

For example, who would have thought that, when antirez stepped down from Redis and the new maintainers promised it would stay OSS, that this would not be the case?

At all points in time since I was added to the previous Redis core team, I thought there was a realistic chance that Redis would decide to relicense the core, because they had the right to do so and their leadership could believe it was in their interest. I obviously didn't know it would happen when it did, but I think people should be distrustful when a single entity controls an OSS project the way Redis did. The skepticism is healthy.

yawaramin commented 7 months ago

Yes, and skepticism is healthy no matter which entity, individual, or committee controls a project. When they have the legal mechanism to do so, that means the risk of relicensing exists. Despite how great the technical leadership committee is today, it's prudent to think about a future where the leadership could change without anyone really noticing or foreseeing it. People don't say 'My driver is great, so I don't need to wear a seat belt', right?