fwessels
commented
8 years ago The limit of 16 nodes is not a 'hard' limit, in fact you can increase it (see here for the code: https://github.com/minio/dsync/blob/master/dsync.go#L50). However due to the fact that all servers are connected to each other, the number of connections scales up significantly and also a larger number of messages is sent out for each lock request (albeit going up just linearly).
We chose 16 for minio object storage out of operational experience and thus this is a fine and natural limit for dsync in our case.
Eg for minio we would rather have users use two clusters with 16 servers each instead of one cluster with 32 servers, all the more if you scale up further to for instance 10 clusters -- in this case it is much easier to replace a (single) badly performing cluster of 16 servers instead of having to figure out why your 160 server cluster is sometimes behaving poorly. And in many cases you can argue that there is little benefit to having for instance a larger 32 system cluster with a single bucket compared to two smaller 16 system clusters with two buckets in total. So we try to preach "simple and small systems scale" and solve scalability this way.
Having said all this, you are free to experiment with a higher limit and you will probably get away with doubling or quadrupling the limit (depending on overall system performance in terms of CPU & networking speed). However if you want to scale it up to 512 or 1024 then it will crawl to an (in practice) unusable speed, if fact then you may want to experiment with the following idea: https://github.com/minio/dsync#scale-beyond-16-nodes (we have not tested this in practice, but it may work in certain use cases).
Hope this answers the question, otherwise if not please let us know.
nvartolomei
harshavardhana
Would you mind to document the shortcomings that limits scalability to 16 nodes? ⚡️ also how this number was figured out