search

sscarduzio / elasticsearch-readonlyrest-plugin

Free Elasticsearch security plugin and Kibana security plugin: super-easy Kibana multi-tenancy, Encryption, Authentication, Authorization, Auditing
https://readonlyrest.com
GNU General Public License v3.0
952 stars 165 forks source link

NullPointerException in cluster k8s install #411

Closed remche closed 2 years ago

remche commented 5 years ago

ror version : 1.16.34 es version : 6.6.0 configuration : 3 nodes cluster

I have a three nodes cluster, running in kubernetes. When I run an image with ror installed, a NullPointerException is throwed by nodes, and the cluster is instable.

I tried with a minimal configuration

readonlyrest.yml :

      access_control_rules:
      - name: "ALL MONITOR"
        type: allow
        actions: ["cluster:monitor/*", "indices:admin/get"]

stack trace :

java.util.concurrent.CompletionException: java.lang.NullPointerException
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
    at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683)
    at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
    at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
    at tech.beshu.ror.acl.ACL.check(ACL.java:204)
    at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:159)
    at tech.beshu.ror.es.IndexLevelActionFilter.lambda$apply$1(IndexLevelActionFilter.java:135)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:131)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:719)
    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:741)
    at org.elasticsearch.rest.action.admin.cluster.RestClusterHealthAction.lambda$prepareRequest$0(RestClusterHealthAction.java:81)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97)
    at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$5(ReadonlyRestPlugin.java:193)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
    at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
    at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
    at tech.beshu.ror.es.RequestInfo.extractRemoteAddress(RequestInfo.java:447)
    at tech.beshu.ror.acl.ACL$1.getRemoteAddress(ACL.java:363)
    at tech.beshu.ror.requestcontext.RequestContext.toString(RequestContext.java:453)
    at tech.beshu.ror.requestcontext.RequestContext.toString(RequestContext.java:374)
    at java.base/java.lang.String.valueOf(String.java:2951)
    at java.base/java.lang.StringBuilder.append(StringBuilder.java:168)
    at tech.beshu.ror.acl.ACL.doLog(ACL.java:170)
    at tech.beshu.ror.acl.ACL.lambda$check$4(ACL.java:217)
    at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
    ... 68 more
sscarduzio commented 5 years ago

Check this:

https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/pull/407#discussion_r254695604

remche commented 5 years ago

Good catch, I'm using a reverse proxy. I will give 1.17.0-pre1 a try and keep you informed.

remche commented 5 years ago

Tried with 1.17.0-pre1, and playing with traefik to preserve source host, but it's always throwing exceptions...

sscarduzio commented 5 years ago

See this comment: he tweaks the nginx so it doesn't close the connection. https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/pull/407#discussion_r254771720

remche commented 5 years ago

I saw it, and tried with ingress.kubernetes.io/custom-request-headers: Connection:keep-alive with no more success...

remche commented 5 years ago

https://github.com/remche/elasticsearch-readonlyrest-plugin/blob/d577154215f72671eac965ad0c601cca166c6e27/es63x/src/main/java/tech/beshu/ror/es/RequestInfo.java#L452

Printing remoteHost returns me internal k8s traefik IP or Cerebro IP... Note that it seems to work correctly, but output is dirty... Tell me if you want me further debug.

sscarduzio commented 5 years ago

we could make the extraction of the OA a lazy var and if you don't use it it won't even try.

sscarduzio commented 5 years ago

@coutoPL is the hostname extraction lazy in the new core?

coutoPL commented 5 years ago

@sscarduzio the RequestInfo class didn't change. Could you explain me what is the problem with current solution, because I don't think I get it?

sscarduzio commented 5 years ago

So, apparently under certain conditions (i.e. reverse proxy closes the connection too early) the source IP address can't be retrieved from the connection. Now, because the we eagerly extract all the data from the RequestInfo in order to compose a RequestContext, even if your ACL doesn't have the hosts rule, you will get the NullPointerException. The proposal is to mitigate this exception making the RequestContext#getRemoteAddress method lazy.

remche commented 5 years ago

After consideration, I'm not sure it's the reverse proxy that causes the excetpion. It might be the kubernetes readinessProbe. The effect and the solution are the same, though.

coutoPL commented 5 years ago

Don't you think this solution just move the problem a little further? Users which doesn't use hosts rule, won't be affected, but the one who uses it will have the same problem. Maybe we should make the remote address optional? Hosts rule is going to return NO_MATCH when remote address is none.

sscarduzio commented 5 years ago

move the problem a little further

Yes it does move a problem we can't fix as far as possible especially from users that are not concerned with remote addresses at all. I think this should be done.

Hosts rule is going to return NO_MATCH when remote address is none

This would be very confusing, as the remote address in a connection should be always known. So I agree. Yes: we should return NO_MATCH; and yes we could print out very explicit warnings instead of a wild NPE.

coutoPL commented 5 years ago

ok, piece of cake

coutoPL commented 5 years ago

I've changed it in new core. See: https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/pull/412/commits/f80dcdfe9d359ca41f6feae3cf6bbc97ae0953fd

sscarduzio commented 5 years ago

https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/blob/f80dcdfe9d359ca41f6feae3cf6bbc97ae0953fd/corex/src/main/scala/tech/beshu/ror/acl/blocks/rules/HostsRule.scala#L61

This should be warn (and exclamation mark too! :P )