search

spring-cloud / spring-cloud-gateway

An API Gateway built on Spring Framework and Spring Boot providing routing and more.
http://cloud.spring.io
Apache License 2.0
4.5k stars 3.31k forks source link

Ratelimiter not working with aws elasticache redis 7.1 #3379

Closed sastie-rai closed 4 months ago

sastie-rai commented 4 months ago

Describe the bug When trying to use the rate limiter with elasticache serverless redis (7.1) the following error occurs:

2024-04-29T21:10:38.653-05:00 DEBUG [api-gateway,,] 5553 --- [api-gateway] [ioEventLoop-5-1] [ ] o.s.c.g.f.ratelimit.RedisRateLimiter : Error calling rate limiter lua

org.springframework.data.redis.RedisSystemException: Error in execution at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:52) at org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection.lambda$translateException$0(LettuceReactiveRedisConnection.java:242) at reactor.core.publisher.Flux.lambda$onErrorMap$27(Flux.java:7267) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:256) at io.lettuce.core.RedisPublisher$ImmediateSubscriber.onError(RedisPublisher.java:895) at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:716) at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:357) at io.lettuce.core.RedisPublisher$SubscriptionCommand.onError(RedisPublisher.java:801) at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnComplete(RedisPublisher.java:761) at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:65) at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63) at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:745) at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:680) at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:597) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.lettuce.core.RedisCommandExecutionException: ERR This Redis command is not allowed from script script: 775635cca91092477cbcee85fd800accb6adc4b2, on @user_script:17. at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147) at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:116) ... 31 common frames omitted

sastie-rai commented 4 months ago

This is using spring-cloud-gateway 4.1.3 on java 21 and spring boot 3.2.5

spencergibb commented 4 months ago

If I'm reading the error correctly, this is the line in question https://github.com/spring-cloud/spring-cloud-gateway/blob/main/spring-cloud-gateway-server%2Fsrc%2Fmain%2Fresources%2FMETA-INF%2Fscripts%2Frequest_rate_limiter.lua#L17

I could be wrong though. I'm not sure what we could do any differently. @mp911de thoughts?

sastie-rai commented 4 months ago

@spencergibb I did some digging, KEYS must be the culprit. elasticache serverless redis makes KEYS unavailable: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/SupportedCommands.html#RestrictedCommandsRedis they only allow SCAN

spencergibb commented 4 months ago

I suppose we document the required redis commands

sastie-rai commented 4 months ago

@spencergibb, looking at the lua file you linked, the only unavailable command in AWS elasticache redis (serverless) is KEYS. So until they decide to support KEYS, this is dead in the water.

spencergibb commented 4 months ago

I don't know redis well enough to know if there is a replacement

sastie-rai commented 4 months ago

Lua scripting within Redis (e.g., via EVAL commands) doesn't natively support iterative commands like SCAN because scripts must run atomically without interruption. Directly adapting KEYS to SCAN inside a Lua script executed in Redis isn’t feasible because you can't persistently store state (like a cursor) between script executions, and scripts should not run non-deterministic commands like SCAN that might return different results on each execution.

I will have to find a different backend for the rate limiter.

spencergibb commented 4 months ago

See #2955, bucket4j has multiple persistence options

spring-cloud-issues commented 4 months ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

spring-cloud-issues commented 4 months ago

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.