health check errors after a while

[Aazhar]

Hello

We try to use the healthcheck endpoint, but after a dozen of requests , this error appears :

{
    "HealthCheck": {
        "error": {
            "message": "Platform constant error code: ENOMEM Cannot allocate memory (12)",
            "stack": [
                "org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:114)",
                "org.lmdbjava.Env$Builder.open(Env.java:458)",
                "org.lmdbjava.Env$Builder.open(Env.java:474)",
                "com.scienceminer.lookup.storage.StorageEnvFactory.getEnv(StorageEnvFactory.java:36)",
                "com.scienceminer.lookup.storage.lookup.OALookup.<init>(OALookup.java:43)",
                "com.scienceminer.lookup.web.healthcheck.LookupHealthCheck.check(LookupHealthCheck.java:38)",
                "com.codahale.metrics.health.HealthCheck.execute(HealthCheck.java:320)",
                "com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:185)",
                "com.codahale.metrics.servlets.HealthCheckServlet.runHealthChecks(HealthCheckServlet.java:149)",
                "com.codahale.metrics.servlets.HealthCheckServlet.doGet(HealthCheckServlet.java:121)",
                "javax.servlet.http.HttpServlet.service(HttpServlet.java:687)",
                "javax.servlet.http.HttpServlet.service(HttpServlet.java:790)",
                "com.codahale.metrics.servlets.AdminServlet.service(AdminServlet.java:108)",
                "javax.servlet.http.HttpServlet.service(HttpServlet.java:790)",
                "io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)",
                "org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)",
                "io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)",
                "io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)",
                "org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)",
                "org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)",
                "org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)",
                "org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)",
                "org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)",
                "org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)",
                "org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)",
                "org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)",
                "org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)",
                "io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)",
                "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)",
                "io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:67)",
                "org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)",
                "org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)",
                "org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)",
                "org.eclipse.jetty.server.Server.handle(Server.java:502)",
                "org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)",
                "org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)",
                "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)",
                "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)",
                "org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)",
                "org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)",
                "org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)",
                "org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)",
                "org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)",
                "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)",
                "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)",
                "org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)",
                "java.lang.Thread.run(Thread.java:748)"
            ]
        },
        "healthy": false,
        "message": "Platform constant error code: ENOMEM Cannot allocate memory (12)"
    },
    "deadlocks": {
        "healthy": true
    }
}

even though this error appears, the API still work .

Cheers

[kermitt2]

Hi @Aazhar !

I was not able to reproduce the error but I think it would be due to the fact that the healthcheck method open LMDB environments (with handle to all db) for every call, without closing them. As the first one for service is started when the server is launched, the rest of the API will still works.

I pushed a fix where the first opened storage environment is reused in the healthcheck, so it is like calling localhost:8080/service/data.

Ideally it would be nice to have the healthcheck method as a normal endpoint (:8080) and not under the adminstrative port (:8081) Thanks !