search

spring-cloud / spring-cloud-gateway

An API Gateway built on Spring Framework and Spring Boot providing routing and more.
http://cloud.spring.io
Apache License 2.0
4.53k stars 3.32k forks source link

Gateway Retry Filter Causes Gateway to Leak Memory #988

Open dave-fl opened 5 years ago

dave-fl commented 5 years ago

Currently under real world test scenario we are finding that the gateway will begin to leak after approximately 20 minutes while under load.

From my testing it seems like the leak will only appear if the retry filter is being used. No retry filter, no leaks. The config we are using for the retry filter is:

.retry(retryConfig -> retryConfig.setRetries(DEFAULT_RETRIES)
    .setExceptions(RetryableHttpClientErrorException.class, IOException.class, TimeoutException.class)
    .setStatuses(HttpStatus.UNAUTHORIZED)
    .setMethods(HttpMethod.GET, HttpMethod.POST))

There have been other comments regarding gateway memory leaks and those users have reported that they are only seeing the leaks when the retry filter is enabled.

Stack traces are the same as what has been seen here:

https://github.com/spring-cloud/spring-cloud-gateway/issues/810

I have tried to make a reproducible example but am currently blocked by the errors coming up here:

https://github.com/spring-cloud/spring-cloud-gateway/issues/982

My latest test was done using the following.

Spring Boot 2.1.3 Release Greenwich.SR1 Netty 4.1.34.Final Californium-SR6 Spring Framework 5.1.6.BUILD-SNAPSHOT

yechao4j commented 5 years ago

hi, dave-fl in my project i was not using RetryGatewayFilterFactory, and also cause LEAK: ByteBuf.release() was not called，so i think there must be any other reason cause this bug, but i can't find where it come from