search

opensearch-project / security-dashboards-plugin

🔐 Manage your internal users, roles, access control, and audit logs from OpenSearch Dashboards
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
70 stars 152 forks source link

[BUG] Kibana throws errors 500/401 one hour after login when using SAML #828

Closed GuiTeK closed 7 months ago

GuiTeK commented 3 years ago

Describe the bug 1 hour after login, Kibana will show one of these 2 errors instead of the requested page:

The only way to work around this issue is to delete cookies.

To Reproduce Steps to reproduce the behavior:

  1. Login to Kibana via a SAML Identity Provider (e.g. Okta)
  2. Wait for 1 hour
  3. Try to refresh/browse to a new Kibana page
  4. See that Kibana shows {"statusCode":401,"error":"Unauthorized","message":"Response Error"} instead of showing the requested page
  5. See that Kibana shows {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."} when manually going to the root Kibana domain

Note: to create the Okta app, I followed the instructions here: AWS - Add Single Sign-On (SSO) to Open Distro for Elasticsearch Kibana using SAML and Okta.

Expected behavior The internal JWT created by OpenDistro (I'm not sure exactly what component creates it) should be automatically renewed and Kibana shouldn't throw an error (either 401 or 500) when visiting a page 1 hour or more after initial login.

Logs ES Node Logs

[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Rest authentication request from 10.0.3.4:32528 [original: /10.0.3.4:32528]
[2021-09-15T09:08:24,861][DEBUG][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Check authdomain for rest noop/0 or 2 in total
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Try to extract auth creds from clientcert http authenticator
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.h.HTTPClientCertAuthenticator] [es3.logs.example.com] No CLIENT CERT, send 401
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] No 'Authorization' header, send 403
[2021-09-15T09:08:24,861][DEBUG][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Check authdomain for rest noop/1 or 2 in total
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Try to extract auth creds from saml http authenticator
[2021-09-15T09:08:24,862][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [es3.logs.example.com] Extracting JWT token from [REDACTED JWT TOKEN HERE] failed
com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: The token has expired
    at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:85) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:108) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.access$000(AbstractHTTPJwtAuthenticator.java:47) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:90) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:87) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
    at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:87) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.extractCredentials(HTTPSamlAuthenticator.java:148) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:421) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.checkAndAuthenticateRequest(OpenDistroSecurityRestFilter.java:177) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.access$000(OpenDistroSecurityRestFilter.java:66) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:113) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]
    at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opendistro_security-1.13.1.0.jar:1.13.1.0]
    at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]
    at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]
    at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]
    at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]
    at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
    at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.apache.cxf.rs.security.jose.jwt.JwtException: The token has expired
    at org.apache.cxf.rs.security.jose.jwt.JwtUtils.validateJwtExpiry(JwtUtils.java:58) ~[cxf-rt-rs-security-jose-3.4.0.jar:3.4.0]
    at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.validateClaims(JwtVerifier.java:119) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
    at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:81) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
    ... 74 more
[2021-09-15T09:08:24,865][DEBUG][c.o.s.a.AuthnRequest     ] [es3.logs.example.com] AuthNRequest --> <samlp:AuthnRequest xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" ID="ONELOGIN_{SOME_UUID}" Version="2.0" IssueInstant="2021-09-15T09:08:24Z" ForceAuthn="true" Destination="https://subdomain.okta.com/app/xxx/yyy/sso/saml" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" AssertionConsumerServiceURL="https://kb.logs.example.com/_opendistro/_security/saml/acs"><saml:Issuer>logs-kibana-saml</saml:Issuer><samlp:NameIDPolicy Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified" AllowCreate="true" /></samlp:AuthnRequest>
[2021-09-15T09:08:24,865][TRACE][o.e.i.b.in_flight_requests] [es3.logs.example.com] [in_flight_requests] Adjusted breaker by [0] bytes, now [0]
[2021-09-15T09:08:24,866][TRACE][o.e.h.HttpTracer         ] [es3.logs.example.com] [422][8027f54c-0e8c-4318-b8c8-a5fab657dd63][UNAUTHORIZED][text/plain; charset=UTF-8][0] sent response to [Netty4HttpChannel{localAddress=/10.0.5.8:9200, remoteAddress=/10.0.3.4:32528}] success [true]
[2021-09-15T09:08:24,866][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

Kibana Server Logs

Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"log","@timestamp":"2021-09-14T15:33:17Z","tags":["error","elasticsearch","data"],"pid":493,"message":"[ResponseError]: Response Error"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"log","@timestamp":"2021-09-14T15:33:17Z","tags":["error","http"],"pid":493,"message":"{ ResponseError: Response Error\n    at IncomingMessage.response.on (/us
r/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/proc
ess/next_tick.js:63:19)\n  name: 'ResponseError',\n  meta:\n   { body: '',\n     statusCode: 401,\n     headers:\n      { 'x-opaque-id': '{SOME_UUID}',\n        'www-authenticate':\n         'X-S
ecurity-IdP realm=\"Open Distro Security\" location=\"https://subdomain.okta.com/app/xxx/yyy/sso/saml?SAMLRequest=some-base64-data\" requestId=\"ONELOGIN_{SOME_UUID}\"',\n        'content-type': 'text/plain; charset=UTF-8',\n        'content-length': '0' },\n
    meta:\n      { context: null,\n        request: [Object],\n        name: 'elasticsearch-js',\n        connection: [Object],\n        attempts: 0,\n        aborted: false } },\n  isBoom: true,\n  isServer: false,\n  d
ata: null,\n  output:\n   { statusCode: 401,\n     payload:\n      { statusCode: 401,\n        error: 'Unauthorized',\n        message: 'Response Error' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedOb
jectsClientErrorCode)]: 'SavedObjectsClient/notAuthorized' }"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"error","@timestamp":"2021-09-14T15:33:17Z","tags":[],"pid":493,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Inter
nal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34
)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/","path":
"/","href":"/"},"message":"Internal Server Error"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"response","@timestamp":"2021-09-14T15:33:17Z","tags":[],"pid":493,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"x-forwarded-for"
:"38.103.45.2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kb.logs.example.com","x-amzn-trace-id":"Root=some-id","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v
=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko
) Chrome/93.0.4577.63 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-m
ode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7"},"remoteAddress":"10.0.1.36","userAgent":"Mozilla/5.0 (Macin
tosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"},"res":{"statusCode":500,"responseTime":56,"contentLength":9},"message":"GET / 500 56ms - 9.0B"}

Host/Environment (please complete the following information):

opendistro-alerting               1.13.1.0-1 
opendistro-anomaly-detection      1.13.0.0-1
opendistro-asynchronous-search    1.13.0.1-1
opendistro-index-management       1.13.2.0-1
opendistro-job-scheduler          1.13.0.0-1
opendistro-knn                    1.13.0.0-1
opendistro-knnlib                 1.13.0.0
opendistro-performance-analyzer   1.13.0.0-1
opendistro-reports-scheduler      1.13.0.0-1
opendistro-security               1.13.1.0-1
opendistro-sql                    1.13.2.0-1
opendistroforelasticsearch        1.13.2-1