Netflix / dynomite

A generic dynamo implementation for different k-v storage engines
Apache License 2.0
4.2k stars 533 forks source link

OOM on first nodes in racks #697

Open rmalenko opened 5 years ago

rmalenko commented 5 years ago

Hello,

We have OOM on first nodes in racks. Our configuration is 3 nodes each has 3 GB RAM for Redis in one rack and we have 2 racks.

Rack - 001    Rack – 002    Node token   Min data token  Max data token
-----------------------------------------------------------------------
dc01-r01-01   dc01-r02-01   0            0               1431655764
dc01-r01-02   dc01-r02-02   1431655765   1431655765      2863311529
dc01-r01-03   dc01-r02-03   2863311530   2863311530      4294967295

When we insert data these nodes become mirrored dc01-r01-01 and dc01-r02-01 We can insert a lot of data until these nodes will use all of allowed RAM and responds OOM but the number of keys is no more 1000

What we have. Two nodes without free RAM dc01-r01-01 and dc01-r02-01 with about 1000 keys and all other nodes with 100% free RAM allowed for Redis and without keys.

Did I rightly understand that Dynamo doesn't take into account RAM and doesn't distribute keys to other nodes because our number of keys is less than 1431655764?

Could you recommend a configuration, please?

BR