search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.32k stars 2.97k forks source link

Trino oauth `Received bad response from userinfo endpoint` , 'Caused by: java.io.EOFException: ' in the HttpConnectionOverHTTP #9771

Open nelaaro opened 2 years ago

nelaaro commented 2 years ago

Hi

I am seeing the following errors on trino oauth, using dex as a backend oauth / ocide / jwt provider. This works some of the time but ends up failing if left running for a long time.

TLDR

2021-10-26T07:06:44.443Z ERROR http-worker-123 io.trino.server.security.oauth2.OAuth2Service Received bad response from userinfo endpoint java.io.UncheckedIOException: Failed communicating with server: https://dex.ourdomain.net/userinfo Caused by: java.io.EOFException: HttpConnectionOverHTTP@b197f84::DecryptedEndPoint@1145b7a6{l=/10.243.2.202:39636,r=dex.ourdomain.net/10.241.26.89:443,OPEN,fill=-,flush=-,to=5014/60000}

If I restart the co-orindator pod things work fine again and logs work. Other wise I see trino with login failed on the redirect page.

Detailed with additional configs and logs at the end.

Trino co-ordinator configs

kubectl-neat get -- -n trino configmap/trino-coordinator -o yaml
apiVersion: v1
data:
access-control.properties: |
    access-control.name=file
    security.refresh-period=30s
    security.config-file=/etc/trino/access-control/rules.json
config.properties: |
    coordinator=true
    node-scheduler.include-coordinator=false
    # query.max-memory=23GB
    # query.max-memory-per-node=24GB
    # query.max-total-memory=26GB
    # query.max-total-memory-per-node=28GB
    # discovery-server.enabled=true
    discovery.uri=http://localhost:8080
    # http-server.https.port=8443
    http-server.http.port=8080
    # https://trino.io/docs/current/security/tls.html#https-load-balancer
    # http-server.authentication.allow-insecure-over-http=true
    # http-server.https.enabled=false
    # http-server.https.enabled=true
    http-server.process-forwarded=true
    http-server.admin.enabled=true
    http-server.authentication.type=PASSWORD,OAUTH2
    web-ui.authentication.type=oauth2
    http-server.authentication.oauth2.issuer=https://dex.ourdomain.net/
    http-server.authentication.oauth2.auth-url=https://dex.ourdomain.net/auth
    http-server.authentication.oauth2.token-url=https://dex.ourdomain.net/token
    http-server.authentication.oauth2.jwks-url=https://dex.ourdomain.net/keys
    http-server.authentication.oauth2.userinfo-url=https://dex.ourdomain.net/userinfo
    # http-server.authentication.oauth2.additional-audiences=
    http-server.authentication.oauth2.scopes=openid,email,groups,profile
    http-server.authentication.oauth2.principal-field=preferred_username
    http-server.authentication.oauth2.user-mapping.pattern=(.*)
    http-server.authentication.oauth2.client-id=trino-auth
    http-server.authentication.oauth2.client-secret=redacted
    # monitoring
    jmx.rmiregistry.port=9080
    jmx.rmiserver.port=9081
group-provider.properties: |
    group-provider.name=file
    file.group-file=/etc/trino/access-control/group.txt
    file.refresh-period=30s
jvm.config: |
    -server
    -XX:MaxRAMPercentage=90
    -XX:MinRAMPercentage=70
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+UseGCOverheadLimit
    -XX:+ExplicitGCInvokesConcurrent
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+ExitOnOutOfMemoryError
    -Djdk.attach.allowAttachSelf=true

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.local.only=false
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.port=9080
    -Dcom.sun.management.jmxremote.rmi.port=9081
log.properties: |
    io.trino=INFO
    io.trinosql=INFO
    #io.trino=DEBUG
    io.trino.server.security.oauth2=DEBUG
node.properties: |
    node.environment=production
    node.data-dir=/data/trino
    plugin.dir=/usr/lib/trino/plugin
password-authenticator.properties: |
    password-authenticator.name=file
    file.password-file=/etc/trino/auth/password.db
    file.refresh-period=30s
kind: ConfigMap
metadata:
labels:
    app.kubernetes.io/component: coordinator
    app.kubernetes.io/instance: trino
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: trino
    app.kubernetes.io/version: "362"
    helm.sh/chart: trino-1.3.2
name: trino-coordinator
namespace: trino

Relevant config oauth logs

These exceptions are happening with oauth logins working. But after some times any login attempt ends with these failures, and failed 

2021-10-26T07:02:24.422Z    INFO    main    Bootstrap   oauth2-jwk.http-client.https.automatic-shared-secret                [REDACTED]                                                                 [REDACTED]
2021-10-26T07:02:24.422Z    INFO    main    Bootstrap   oauth2-jwk.http-client.use-blocking-connect                         false                                                                      false
2021-10-26T07:02:24.422Z    INFO    main    Bootstrap   oauth2-jwk.http-client.connect-timeout                              5.00s                                                                      5.00s
2021-10-26T07:02:24.425Z    INFO    main    Bootstrap   oauth2-jwk.http-client.http2.enabled                                false                                                                      false                                                                      Enable the HTTP/2 transport
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.http2.session-receive-window-size            16MB                                                                       16MB                                                                       Initial size of session's flow control receive window for HTTP/2
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.http2.stream-receive-window-size             16MB                                                                       16MB                                                                       Initial size of stream's flow control receive window for HTTP/2
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.http2.input-buffer-size                      8kB                                                                        8kB                                                                        Size of the buffer used to read from the network for HTTP/2
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.https.excluded-cipher                        [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]  [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]  Setting this config property overwrites Jetty's default excluded cipher suites
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.https.included-cipher                        []                                                                         []
2021-10-26T07:02:24.426Z    INFO    main    Bootstrap   oauth2-jwk.http-client.idle-timeout                                 1.00m                                                                      1.00m
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.keep-alive-interval                          ----                                                                       ----
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.key-store-password                           [REDACTED]                                                                 [REDACTED]
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.key-store-path                               ----                                                                       ----
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.buffer-size                              1MB                                                                        1MB
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.compression.enabled                      true                                                                       true
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.enabled                                  false                                                                      false
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.flush-interval                           10.00s                                                                     10.00s
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.max-history                              15                                                                         15
2021-10-26T07:02:24.427Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.max-size                                 1GB                                                                        1GB
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.path                                     var/log/                                                                   var/log/                                                                   The name of the log file will be prefixed with the name of the HTTP client (<client_name>-http-client.log)
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.log.queue-size                               10000                                                                      10000
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.max-connections                              200                                                                        200
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.max-connections-per-server                   20                                                                         20
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.max-content-length                           16MB                                                                       16MB
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.max-requests-queued-per-destination          1024                                                                       1024
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.max-threads                                  200                                                                        200
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.min-threads                                  8                                                                          8
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.record-request-complete                      true                                                                       true
2021-10-26T07:02:24.428Z    INFO    main    Bootstrap   oauth2-jwk.http-client.request-buffer-size                          4kB                                                                        4kB
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.request-timeout                              5.00m                                                                      5.00m
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.response-buffer-size                         16kB                                                                       16kB
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.secure-random-algorithm                      ----                                                                       ----
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.selector-count                               2                                                                          2
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.socks-proxy                                  ----                                                                       ----
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.timeout-concurrency                          1                                                                          1                                                                          Number of concurrent locks for timeout
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.timeout-threads                              1                                                                          1                                                                          Total number of timeout threads
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.trust-store-password                         [REDACTED]                                                                 [REDACTED]
2021-10-26T07:02:24.429Z    INFO    main    Bootstrap   oauth2-jwk.http-client.trust-store-path                             ----                                                                       ----
2021-10-26T07:02:24.430Z    INFO    main    Bootstrap   oauth2-jwk.http-client.https.hostname-verification                  true                                                                       true                                                                       Verify that server hostname matches the server certificate

2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.allow-insecure-over-http                 false                                                                      false                                                                      Insecure authentication over HTTP (non-secure) enabled
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.access-token-issuer               ----                                                                       ----                                                                       The required issuer for access tokens
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.additional-audiences              []                                                                         []                                                                         Additional audiences to trust in addition to the Client ID
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.auth-url                          ----                                                                       https://dex.ourdomain.net/auth                                         URL of the authorization server's authorization endpoint
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.challenge-timeout                 15.00m                                                                     15.00m                                                                     Maximum duration of OAuth2 authorization challenge
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.client-id                         ----                                                                       trino-auth                                                                 Client ID
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.client-secret                     [REDACTED]                                                                 [REDACTED]                                                                 Client secret
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.issuer                            ----                                                                       https://dex.ourdomain.net/                                             The required issuer of a token
2021-10-26T07:02:24.463Z    INFO    main    Bootstrap   http-server.authentication.oauth2.jwks-url                          ----                                                                       https://dex.ourdomain.net/keys                                         URL of the authorization server's JWKS (JSON Web Key Set) endpoint
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.principal-field                   sub                                                                        preferred_username                                                         The claim to use as the principal
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.scopes                            [openid]                                                                   [openid, email, groups, profile]                                           Scopes requested by the server during OAuth2 authorization challenge
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.state-key                         ----                                                                       ----                                                                       A secret key used by HMAC algorithm to sign the state parameter
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.token-url                         ----                                                                       https://dex.ourdomain.net/token                                        URL of the authorization server's token endpoint
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.user-mapping.file                 ----                                                                       ----                                                                       File containing rules for mapping user
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.user-mapping.pattern              ----                                                                       (.*)                                                                       Regex to match against user name
2021-10-26T07:02:24.464Z    INFO    main    Bootstrap   http-server.authentication.oauth2.userinfo-url                      ----                                                                       https://dex.ourdomain.net/userinfo                                     URL of the userinfo endpoint

Errors at log in attempts / stack trace 

2021-10-26T07:06:12.819Z    INFO    http-worker-241 io.trino.plugin.base.security.FileBasedSystemAccessControl  Refreshing system access control from /etc/trino/access-control/rules.json
2021-10-26T07:06:44.443Z    ERROR   http-worker-123 io.trino.server.security.oauth2.OAuth2Service   Received bad response from userinfo endpoint
java.io.UncheckedIOException: Failed communicating with server: https://dex.ourdomain.net/userinfo
at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22)
at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62)
at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:566)
at io.trino.server.security.oauth2.OAuth2Service.internalConvertTokenToClaims(OAuth2Service.java:329)
at io.trino.server.security.oauth2.OAuth2Service.convertTokenToClaims(OAuth2Service.java:315)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.getAccessToken(OAuth2WebUiAuthenticationFilter.java:119)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.filter(OAuth2WebUiAuthenticationFilter.java:84)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: java.io.EOFException: HttpConnectionOverHTTP@b197f84::DecryptedEndPoint@1145b7a6{l=/10.243.2.202:39636,r=dex.ourdomain.net/10.241.26.89:443,OPEN,fill=-,flush=-,to=5014/60000} at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1614) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) ... 3 more

2021-10-26T07:10:51.792Z    INFO    http-worker-278 io.trino.plugin.base.security.FileBasedSystemAccessControl  Refreshing system access control from /etc/trino/access-control/rules.json
2021-10-26T07:11:07.033Z    ERROR   http-worker-126 io.trino.server.security.oauth2.OAuth2Service   Received bad response from userinfo endpoint

j ava.io.UncheckedIOException: Failed communicating with server: https://dex.ourdomain.net/userinfo at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22) at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62) at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:566) at io.trino.server.security.oauth2.OAuth2Service.internalConvertTokenToClaims(OAuth2Service.java:329) at io.trino.server.security.oauth2.OAuth2Service.convertTokenToClaims(OAuth2Service.java:315) at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.getAccessToken(OAuth2WebUiAuthenticationFilter.java:119) at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.filter(OAuth2WebUiAuthenticationFilter.java:84) at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132) at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68) at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.io.EOFException: HttpConnectionOverHTTP@1bf6a5ca::DecryptedEndPoint@365244e1{l=/10.243.2.202:37576,r=dex.ourdomain.net/10.243.13.68:443,OPEN,fill=-,flush=-,to=5105/60000} at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1614) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ... 10 more

2021-10-26T07:13:31.198Z    ERROR   http-worker-287 io.trino.server.security.oauth2.OAuth2Service   Received bad response from userinfo endpoint
java.io.UncheckedIOException: Failed communicating with server: https://dex.ourdomain.net/userinfo
at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22)
at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62)
at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:566)
at io.trino.server.security.oauth2.OAuth2Service.internalConvertTokenToClaims(OAuth2Service.java:329)
at io.trino.server.security.oauth2.OAuth2Service.convertTokenToClaims(OAuth2Service.java:315)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.getAccessToken(OAuth2WebUiAuthenticationFilter.java:119)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.filter(OAuth2WebUiAuthenticationFilter.java:84)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.EOFException: HttpConnectionOverHTTP@11265c45::DecryptedEndPoint@aeabc83{l=/10.243.2.202:48302,r=dex.ourdomain.net/10.242.34.53:443,OPEN,fill=-,flush=-,to=5286/60000}
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1614)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
... 10 more

2021-10-26T07:13:32.970Z    INFO    http-worker-241 io.trino.plugin.base.security.FileBasedSystemAccessControl  Refreshing system access control from /etc/trino/access-control/rules.json
2021-10-26T07:13:38.074Z    ERROR   http-worker-287 io.trino.server.security.oauth2.OAuth2Service   Received bad response from userinfo endpoint
java.io.UncheckedIOException: Failed communicating with server: https://dex.ourdomain.net/userinfo
at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22)
at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62)
at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:566)
at io.trino.server.security.oauth2.OAuth2Service.internalConvertTokenToClaims(OAuth2Service.java:329)
at io.trino.server.security.oauth2.OAuth2Service.convertTokenToClaims(OAuth2Service.java:315)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.getAccessToken(OAuth2WebUiAuthenticationFilter.java:119)
at io.trino.server.ui.OAuth2WebUiAuthenticationFilter.filter(OAuth2WebUiAuthenticationFilter.java:84)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: java.io.EOFException: HttpConnectionOverHTTP@317c64d6::DecryptedEndPoint@53ca0bb9{l=/10.243.2.202:48376,r=dex.ourdomain.net/10.242.34.53:443,OPEN,fill=-,flush=-,to=5077/60000} at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1614) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ... 10 more

Complete logs

https://paste-bin.xyz/8773

nelaaro commented 2 years ago

This is deployed in aws with a alb ingress and load balancer. The dex endpoint is sitting behind a nginx ingress and aws nlb load balancer.

I think shorting the time outs would probably fix this will update if I find any useful work arounds

lukasz-walkiewicz commented 2 years ago

Hi @nelaaro,

Thanks for reporting this. Did you verify that it's not network/configuration related? Can you connect to Dex's /userinfo endpoint from the coordinator pod while the coordinator itself is experiencing those problems?

nelaaro commented 2 years ago

Yes, it is an intermittent issue that only affects trino. Trino will work, logins will work and we will see these exceptions. Then after about 2 days it completely stops being able to process requests. While all our other services connecting to dex work without issue. We have a number of other services using the same endpoints without issues.

From what I could google. https://docs.oracle.com/javase/tutorial/essential/io/datastreams.html https://stackoverflow.com/questions/50395175/http-request-failing-with-eofexception

It could be that the end of the HTTP stream exception is not being caught and handled properly for this one endpoint. if dex was an issue I would expect exceptions across the board on all the other endpoints as well. This could also be a timing issue related to keeping alive and timeouts configured in the jwks, or OAuth components.

There a couple of options (undocumented) that I am go to try out tomorrow to see how they affect things. oauth2-jwk.http-client.use-blocking-connect oauth2-jwk.http-client.keep-alive-interval oauth2-jwk.http-client.max-connections
oauth2-jwk.http-client.max-connections-per-server

I am thinking this could be related to the time-outs on the load balancers coming into trino AWS alb and then dex AWS nlb. And the timeouts configured on those services.
I will be running some packet dumps tomorrow to see what I can figure out.

lukasz-walkiewicz commented 2 years ago

Thanks, for the detailed response.

I will be running some packet dumps tomorrow to see what I can figure out.

That would be great.

nelaaro commented 2 years ago

Due to our setup, we have an AWS certificate managed by ACM used by our load balancers. We don't have access to the private key to decrypt the SSL traffic in the traffic dump.

kubectl sniff trino-coordinator-7ff46bd59b-fnf6t -c trino-coordinator -n trino -p

Does work and I am able to get a packet dump but the interesting traffic exists in the https encrypted traffic. I am going to look at the basic TCP control sequences to see if I can identify in strange TCP traffic issues. This is made harder by not seeing the exact HTTP contents in the TLS traffic.

I am also going to have a look at a test cluster we have and try to install certbot / letsencrypt to work around the https traffic. See if I can replicate the same problems.

What I do notice on replicating these issues. Is that the calls to the userinfo endpoint look to happen every time a request to trino to run a query. So it verifies the user session details/token when running the query. I am going to try the packet capture on the dex side and see if I have some inside there as well.