search

redisson / redisson

Redisson - Easy Redis Java client and Real-Time Data Platform. Valkey compatible. Sync/Async/RxJava/Reactive API. Over 50 Redis or Valkey based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Bloom filter, Spring, Tomcat, Scheduler, JCache API, Hibernate, RPC, local cache...
https://redisson.pro
Apache License 2.0
23.31k stars 5.35k forks source link

RRateLimiter drift #3804

Closed alim-akbashev closed 3 years ago

alim-akbashev commented 3 years ago

Rate limit decreases over the time in highly concurrent environment. Demo of such drift is on screenshot. Rate there has been limited by only RRateLimiter. image

Expected behavior Limit should not change without explicit call of setRate() with new value.

Actual behavior Limit changes over hours. As a workaround I explicitly call setRate() with the same values time to time.

Steps to reproduce or test case Just regular usage, nothing special:

    final long limitPerSecond = 115;
    final RRateLimiter limiter = redisson.getRateLimiter(name);
    limiter.setRate(RateType.OVERALL, limitPerSecond, 1000, RateIntervalUnit.MILLISECONDS);
    // ...
    limiter.acquire();

Redis version 6.0.5

Redisson version 3.16.1

Redisson configuration

    final Config config = new Config();
    config.useSingleServer().setAddress(url);
mrniko commented 3 years ago

Can you try version attached?

redisson-3.16.2-SNAPSHOT.jar.zip

alim-akbashev commented 3 years ago

thank you for a quick response.

i've deployed service with version you've provided, collecting metrics now, will be back with feedback tomorrow

alim-akbashev commented 3 years ago

as I can see, the issue still there: image

by checking values in redis, it seems that after some time a cardinality of sorted set ...:permits stops matching the value ...:value

mrniko commented 3 years ago

Did you make sure that all Redisson instances have synced clock?

alim-akbashev commented 3 years ago

not sure about nanoseconds precision, but i'm sure clocks are pretty in sync. all instances are running in the same k8s cluster (amazon eks), all k8s nodes are synced.

btw, not sure if it's important, my redis is a managed one - single AWS ElastiCache running on m6g.large (64bit arm Graviton).

let me try to build more reproducible sandbox

alim-akbashev commented 3 years ago

it's more or less reproducible. Please check the snippet https://gist.github.com/alim-akbashev/316f56197f099f323ff68cd27b51df78 It's more simple to reproduce dritft when numbers in config are big, like 30k requests per 30 seconds, but issue appears with lower values as well, it just require more time to drift.

just in case, my config i've used to run that snippet: just regular macbook pro (Intel), Redis 6.2.5 in docker, openjdk 16

p.s. while i've been typing this comment, limit has dropped down to 29494, so it takes minutes, not hours with such config

mrniko commented 3 years ago

Fixed! Thanks for report