Memory leak when call ProxyHttpClient.proxy

[vinhbt]

Thanks for reporting an issue, please review the task list below before submitting the issue. Your issue report will be closed if the issue is incomplete and the below tasks not completed.

NOTE: If you are unsure about something and the issue is more of a question a better place to ask questions is on Stack Overflow (https://stackoverflow.com/tags/micronaut) or Gitter (https://gitter.im/micronautfw/). DO NOT use the issue tracker to ask questions.

Task List

• [x] Steps to reproduce provided
• [ ] Stacktrace (if present) provided
• [ ] Example that reproduces the problem uploaded to Github
• [x] Full description of the issue provided (see below)

Steps to Reproduce

1. Create a filter @Filter("/**") public class ProxyFilter extends OncePerRequestHttpServerFilter { @Override protected Publisher<MutableHttpResponse<?>> doFilterOnce(HttpRequest<?> request, ServerFilterChain chain) {

   .....

   client.proxy( request.mutate() .uri(b -> b.scheme("http") .host(route.getHost()) .port(route.getPort()) ) .header(INTERNAL_USER_ID, userUid) .header(EXTERNAL_CALL, "true") } }

2. Memory leak occur

Expected Behaviour

Not memory leak

Environment Information

• Operating System: docker alpine, K8s on EKS
• Micronaut Version: 2.0.0, 2.0.1, 2.0.2
• JDK Version: 11

Example Application

• TODO: link to github repository with example that reproduces the issue

@Filter("/**") public class ProxyFilter extends OncePerRequestHttpServerFilter {

private static final Logger LOG = LoggerFactory.getLogger(ProxyFilter.class);
private static final String INTERNAL_USER_ID = "internal.user_id";
private static final String EXTERNAL_CALL = "external.call";

private final RxProxyHttpClient client;
private final BearerTokenReader bearerTokenReader;
private final JwtTokenValidator jwtTokenValidator;
private final ConfigurationUrlMapRule configurationUrlMapRule;
private final List<ServiceConfig> configRoutes;

public ProxyFilter(RxProxyHttpClient client,
                   ConfigurationUrlMapRule configurationUrlMapRule,
                   ProxyConfigurationProperties proxyConfigurationProperties,
                   BearerTokenReader bearerTokenReader,
                   JwtTokenValidator jwtTokenValidator) {
    this.client = client;
    this.bearerTokenReader = bearerTokenReader;
    this.jwtTokenValidator = jwtTokenValidator;
    this.configurationUrlMapRule = configurationUrlMapRule;
    this.configRoutes = proxyConfigurationProperties.getServiceConfigs();
}

private Flowable<Authentication> getAuthFromRequest(HttpRequest<?> request){
    Optional<String> tokenOpt = bearerTokenReader.findToken(request);
    if (tokenOpt.isPresent()) {
        String tokenValue = tokenOpt.get();
        return Flowable.fromPublisher(jwtTokenValidator.validateToken(tokenValue, request));
    } else {
        return Flowable.empty();
    }
}

@Override
protected Publisher<MutableHttpResponse<?>> doFilterOnce(HttpRequest<?> request, ServerFilterChain chain) {

    return getAuthFromRequest(request)
            .flatMap(authentication -> checkRules(request, chain, authentication))
            .switchIfEmpty(Flowable.defer(() -> checkRules(request, chain, null)));
}

/**
 * Check the security rules against the provided arguments.
 *
 * @param request        The request
 * @param chain          The server chain
 * @param authentication The authentication
 * @return A response publisher
 */
protected Publisher<MutableHttpResponse<?>> checkRules(HttpRequest<?> request,
                                                       ServerFilterChain chain,
                                                       @Nullable Authentication authentication) {
    HttpMethod method = request.getMethod();
    String path = request.getPath();
    Map<String, Object> attributes = authentication != null ? authentication.getAttributes() : null;
    boolean forbidden = authentication != null;

    SecurityRuleResult result = configurationUrlMapRule.check(request, attributes);
    if (result == SecurityRuleResult.REJECTED) {
        if (LOG.isDebugEnabled()) {
            LOG.debug("Unauthorized request {} {}.", method, path);
        }
        request.setAttribute(REJECTION, forbidden ? HttpStatus.FORBIDDEN : HttpStatus.UNAUTHORIZED);
        return Publishers.just(HttpResponse.status(forbidden ? HttpStatus.FORBIDDEN : HttpStatus.UNAUTHORIZED));
    }
    if (result == SecurityRuleResult.ALLOWED) {
        if (LOG.isDebugEnabled()) {
            LOG.debug("Authorized request {} {}", method, path);
        }
        Optional<ServiceConfig> config = configRoutes
                .stream()
                .filter(item -> path.startsWith(item.getPath()))
                .findFirst();

        if (config.isPresent() && config.get().getIsForceProxy()) {
            ServiceConfig route = config.get();
            String userUid = authentication == null ? null : authentication.getName();
            if (userUid != null) {
                return client.proxy(
                        request.mutate()
                                .uri(b ->
                                        b.scheme("http")
                                                .host(route.getHost())
                                                .port(route.getPort())
                                )
                                .header(INTERNAL_USER_ID, userUid)
                                .header(EXTERNAL_CALL, "true")
                );
            } else {
                return client.proxy(
                        request.mutate()
                                .uri(b ->
                                        b.scheme("http")
                                                .host(route.getHost())
                                                .port(route.getPort())
                                )
                                .header(EXTERNAL_CALL, "true")
                );
            }
        }
    }

    if (LOG.isDebugEnabled()) {
        LOG.debug("Authorized request {} {}. No rule provider authorized or rejected the request.", method, path);
    }
    //no rule found for the given request
    return Publishers.just(HttpResponse.notFound());
}

@Override
public int getOrder() {
    return ServerFilterPhase.SECURITY.before();
}

[graemerocher]

Please attach an example that reproduces the issue

[vinhbt]

gateway-service.zip

[graemerocher]

And what are the steps to reproduce with the attached example?

[vinhbt]

Please wait, I will create a docker compose file and a sample service to reproduce this bug

[graemerocher]

that would be a great help, thanks

[vinhbt]

test.zip I write a docker-compose file that can help you to reproduce this bug.

1. run jibDockerBuild gradle's task in 2 project user-service and gateway-service to generate docker image.
2. run docker-compose up
3. call /api/user/test multi time memory leak occur. It is the cause of error "SLF4J: Failed toString() invocation on an object of type [java.lang.OutOfMemoryError]" in my project

Some error log from my server: Exception in thread "RxSchedulerPurge-1" java.lang.OutOfMemoryError: Java heap space 10/1/2020 4:53:15 PM Exception in thread "nioEventLoopGroup-1-1" java.lang.OutOfMemoryError: Java heap space 10/1/2020 4:53:16 PM 2020-10-01T09:53:15.913 WARN i.n.u.c.AbstractEventExecutor - A task raised an exception. Task: io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker@25abdd86 10/1/2020 4:53:16 PM java.lang.OutOfMemoryError: Java heap space 10/1/2020 4:53:16 PM 2020-10-01T09:53:16.749 ERROR i.m.s.DefaultTaskExceptionHandler - Error invoking scheduled task for bean [io.micronaut.management.health.monitor.HealthMonitorTask@4aa8ff83] Java heap space 10/1/2020 4:53:16 PM java.lang.OutOfMemoryError: Java heap space 10/1/2020 4:53:16 PM Exception in thread "nioEventLoopGroup-1-2" java.lang.OutOfMemoryError: Java heap space 10/1/2020 4:53:30 PM Exception in thread "nioEventLoopGroup-1-3" java.lang.OutOfMemoryError: Java heap space

[meredrica]

any news on this? we see the same behaviour with 2.2.0. RAM constantly grows until we run OOM

[graemerocher]

Will investigate, thanks for the reminder

[graemerocher]

@vinhbt I am not able to reproduce with the example attached (I upgraded the sample to 2.2.0), I have tried reducing my docker resources, did you place load on the server with something or how did you call the server? I have run 30-40 curl requests and memory usage is largely stable (fluctuates sometimes but goes back down)

@meredrica if you have another example please feel free to attach to the issue

[vinhbt]

@graemerocher I have tried with 2000 call for my example.

for i in seq 1 2000; do curl --location --request GET 'http://localhost:8080/api/user/test' \n--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI0OTgiLCJuYmYiOjE2MDEzMDAzNzIsInJvbGVzIjpbIlVTRVIiXSwiaXNzIjoibWljcm9uYXV0LXN3YWdnZXIiLCJtb2JpbGUiOiIrODQ1NTU1NTU1NTUiLCJleHAiOjE2MDEzODY3NzIsImlhdCI6MTYwMTMwMDM3Mn0.o3srTjnsYifbIHqmyCzfaHSt7xQZtm1OP9-tUi0an7I';

[graemerocher]

@vinhbt ok will try.. thanks