search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.36k stars 1.3k forks source link

Update the accuracy of statements made in the documentation #1363

Closed alexmiller-apple closed 5 years ago

alexmiller-apple commented 5 years ago

“FoundationDB has been tested with databases up to 100 TB (total size of key-value pairs – required disk space will be significantly higher after replication and overhead).” https://apple.github.io/foundationdb/known-limitations.html#database-size

Should probably not make it sound like FDB only supports databases up to 100TB in size.

We should probably also clarify basically everything quoted in https://news.ycombinator.com/item?id=19400018 to not be misinterpretable.

ajbeamon commented 5 years ago

We should probably also clarify basically everything quoted in https://news.ycombinator.com/item?id=19400018 to not be misinterpretable.

True, it seems one part of our documentation led to this statement about FoundationDB on Fauna's database comparison:

Isolation is violated when atomic operations are interleaved with regular operations on the same key; manual locking can mitigate this.

alexmiller-apple commented 5 years ago

I ended up leaving what is effectively the "100TB of user data" guidance, because with the typical triple replication, overhead, an additional factor to leave extra space on disks for a bit of growth and fluctuations due to data distribution, etc., it will probably be more like 500TB of disks, which is probably what our upper recommendable limit is right now.

Otherwise, I've fixed most of the issues highlighted in the hacker news discussion, but our documentation could still use a lot of love :/

ouvaa commented 4 months ago

@ajbeamon @alexmiller-apple 100TB or 500TB is actually not a lot in today's use. was wondering so does this mean it's not designed for petabyte scale storage etc?

in that case, what is fdb good for? how is it used in apple etc?

p.s. : i'm asking this question because i needed some distributed sql and came across this: https://github.com/losfair/mvsqlite

sqlite has limit of 281TB, but on top of fdb which is recommended at 500TB, i was wondering if i can only have 2 tenants maxing each other out at 250TB each that's all. (hypothetically) 

 tuning with clusters of up to 500 cores/processes

that seems to suggest 1TB per core / process?