dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
24.84k stars 894 forks source link

Dragonfly not rejecting inserts after reaching maxmemory #2984

Open adiholden opened 3 months ago

adiholden commented 3 months ago

How to reproduce: run dragonfly ./dragonfly --alsologtostderr --dbfilename= --port=6379 --maxmemory=350MB --proactor_threads=1

run memtier memtier_benchmark -c 2 -t 4 --pipeline=30 --hide-histogram --test-time=3000 --distinct-client-seed --expiry-range=100-10000 --data-size-range=3000-4000

expect dragonfly to reject inserts after reaching 350MB but memtier continues to run inserting more entries

adiholden commented 2 months ago

it looks like fiber_periodicdone.WaitFor(period_ms) does not return from wait while running memtier

cyppe commented 2 months ago

I can confirm I have had the same issue. It behaves very strange when reaching maxmemory.

It always accepts write, and in many cases it seems like it works fine to also get the data so it actually stored it. But in many other cases it does not store the write. So my health check that was based on writing a key, get the value, and compare it with what I wrote actually passed in most cases but for our users it still did not work.

Might be dependant on the size of data written? If it's very close to hitting the limit so some write is actually stored and some are not. But both without error on SET.

Don't know if this information helps, but I hope so.

romange commented 2 months ago

@cyppe if you use cache_mode=true, then Dragonfly will evict entries once it reaches maxmemory.

cyppe commented 2 months ago

Yeh thanks, but I use it not only for cache so auto eviction is not an option here.

Would be better if it rejects operations similar to redis if noeviction is used and it reached maxmemory.