search

erikgrinaker / toydb

Distributed SQL database in Rust, written as a learning project
Apache License 2.0
5.99k stars 555 forks source link

Curious why this isn't suitable for production use? #70

Closed matthewmueller closed 2 weeks ago

matthewmueller commented 2 weeks ago

Hey @erikgrinaker, really love this repo for learning the internals of databases!

Outside of the obvious reasons (not battle-tested like SQLite, Postgres, MySQL etc.), I'm curious why you consider this not production ready? It seems like toydb has all the fundamental pieces and those pieces seem well-tested.

Are there certain missing features or shortcuts taken that make this project particularly not suitable for production?

erikgrinaker commented 2 weeks ago

Hey! Yeah, I've taken as many shortcuts as possible to keep the system reasonably simple. There's far too many issues to list here, but to mention a few: no horizontal scalability or load balancing, can't add/remove nodes while running, storage compaction rewrites the entire dataset (blocking all operations), no schema change support, full table scans pull the entire table into memory, the Raft and storage engines are single-threaded, no Raft snapshots or log truncation, no Raft pre-vote or checkquorum (for network partition tolerance), etc.

There's tons of stuff like this -- building a real, production-ready distributed SQL database will take a large team several years to pull off. I'm planning to write an article series in a while that goes into this in some more detail.

matthewmueller commented 2 weeks ago

Awesome, thanks for enumerating! One other thing I noticed while digging in was the lack of support for binding variables

Thanks again for creating this library. I don't really know Rust, but have been able to follow along the code quite well.