search

openanalytics / shinyproxy

ShinyProxy - Open Source Enterprise Deployment for Shiny and data science apps
https://www.shinyproxy.io
Apache License 2.0
525 stars 151 forks source link

[Bug Report] AJAX error in the new release of ShinyProxy (version 2.2.0+) #140

Closed shrektan closed 4 years ago

shrektan commented 5 years ago

In the new release version (2.2.0) of ShinyProxy, all the Apps that use DT with server processing mode will break. The browser complains Ajax errors while ShinyProxy log says Request Methods 'Post' not supported.

Screen Shots

屏幕快照 2019-03-28 下午11 24 18 屏幕快照 2019-03-28 下午11 24 32

An example app

library(shiny)
ui <- fluidPage(DT::DTOutput('tbl'))
server <- function(input, output, session) {
  output$tbl <- DT::renderDT(iris, server = TRUE)
}
shinyApp(ui, server)

I actually set up a docker-compose example for this, see https://github.com/shrektan/shinyproxy-docker-compose-example

tverbeke commented 5 years ago

Thanks! We can reproduce this and are working on it.

fmichielssen commented 5 years ago

@shrektan thanks for reporting the issue. A fix has been committed: https://github.com/openanalytics/containerproxy/commit/6455abdaed031297baf600f5e71e2242b689d940

Can you please give this dev build a try and see if it fixes the problem? https://nexus.openanalytics.eu/nexus/content/repositories/snapshots/eu/openanalytics/shinyproxy/2.2.1-SNAPSHOT/shinyproxy-2.2.1-20190329.093240-2.jar

shrektan commented 5 years ago

@fmichielssen Thanks for the quick fix. I can confirm both DT and fileInput() works fine with the dev build.

tverbeke commented 5 years ago

Thanks for the confirmation, @shrektan! Released it as part of ShinyProxy 2.2.1

bogdanrau commented 5 years ago

Just did a fresh install with 2.2.1 and I'm getting the same error as @shrektan reported:

Screen Shot 2019-03-31 at 2 27 42 PM

Then, in shinyproxy.log, I get:

2019-03-31 21:27:33.548 ERROR 9925 --- [XNIO-2 I/O-3] io.undertow.proxy                        : UT005028: Proxy request to /proxy_endpoint/b77c8f8f-8dc9-45c3-a858-e12c7e33e159/session/cf6134effe616504495d1d75171f0546/dataobj/de_main_table failed

io.undertow.server.TruncatedResponseException: null
    at io.undertow.client.http.HttpRequestConduit.truncateWrites(HttpRequestConduit.java:711) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.terminateWrites(AbstractFixedLengthStreamSinkConduit.java:256) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at org.xnio.conduits.ConduitStreamSinkChannel.shutdownWrites(ConduitStreamSinkChannel.java:178) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
    at io.undertow.channels.DetachableStreamSinkChannel.shutdownWrites(DetachableStreamSinkChannel.java:79) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$HTTPTrailerChannelListener.handleEvent(ProxyHandler.java:754) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:646) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:561) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.client.http.HttpClientExchange.invokeReadReadyCallback(HttpClientExchange.java:212) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.client.http.HttpClientConnection.initiateRequest(HttpClientConnection.java:410) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.client.http.HttpClientConnection.sendRequest(HttpClientConnection.java:343) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction.run(ProxyHandler.java:561) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.util.SameThreadExecutor.execute(SameThreadExecutor.java:35) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.HttpServerExchange.dispatch(HttpServerExchange.java:815) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:316) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:290) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyConnectionPool.connectionReady(ProxyConnectionPool.java:338) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:525) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:301) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at eu.openanalytics.containerproxy.util.ProxyMappingManager$1.getConnection(ProxyMappingManager.java:88) [containerproxy-0.8.1.jar!/:0.8.1]
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:310) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
    at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:582) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:466) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]

What's interesting is that this affects another package: shinyWidgets. I have a bunch of inputs that get generated using insertUI, and the values in the dropdown list highlighted below get pulled from a database (dynamic). When running local, everything runs great (and I get the actual values). When deploying to shinyproxy, I get NA1, NA2, NA3...super odd.

Screen Shot 2019-03-31 at 2 30 48 PM

I should mention that things were working fine on 2.1.0. In case this makes a difference, I am using shinyproxy behind nginx, with a LetsEncrypt cert.

tverbeke commented 5 years ago

@bogdanrau Could you provide a dummy example app (similarly to the one @shrektan provided) that allows us to rapidly reproduce?

dylancis commented 5 years ago

Hey @tverbeke I was wondering if we could now deploy 2.2.1 - is it fixed ?

tverbeke commented 5 years ago

Hello @dylancis: the original problem is definitely fixed as confirmed by our tests and by @shrektan. For the issue reported by @bogdanrau, we don't have a reproducible example yet or (confirmation from another user there is still an issue).

bogdanrau commented 5 years ago

@tverbeke, I just tried the sample app that @shrektan provided and I am getting that error. The only difference I can see is that I'm using the .deb as a service, rather than using the .jar file. Is anyone using the .deb file?

dylancis commented 5 years ago

works fine here with 2.2.1.

bogdanrau commented 5 years ago

works fine here with 2.2.1.

And that's using the deb installer, or the jar file? I created a new VM from scratch just using the .deb file and still had the same issues for whatever reason.

shrektan commented 5 years ago

what is the deb file?

bogdanrau commented 5 years ago

@shrektan : if you look at the shinyproxy download page (https://www.shinyproxy.io/downloads/), you can download either a platform-independent file (.jar), or a deb file specific to Ubuntu that you install with dpkg. I chose that vs. the platform independent file.

rschuchmann commented 5 years ago

I get the same error like @bogdanrau. My setup: I am using shinyproxy 2.2.1 (platform-independent .jar) behind nginx. I have noticed that this error occurs depending on the chosen authentication method in ShinyProxy. When I use the example from @shrektan (docker pull shrektan/rdocker4working) together with nginx, ShinyProxy 2.2.1 and authentication: simple, everything works fine. When I switch to authentication: keycloak I get the error. The Keycloak authentication itself seems to work.

bogdanrau commented 5 years ago

Can confirm that I'm also using keycloak with nginx.

billy34 commented 5 years ago

I also have this problem with version >2.1.0 (last known to work) even 2.2.1 does not work for me. I run a dockerized shinyproxy that fetch the platform independent jar behind a Traefik reverse proxy v1.7.12 . I'm using keycloak v6.0.1 for Authentication/Authorization. I had to revert to version 2.1 to be able to see datatables with ajax enabled (server=TRUE) when disabled it works with newer versions.

Diniodoc commented 5 years ago

Are there any news or workarounds for that bug?

I run in exact the same problem and can confirm that the problem occurs in combination with keycloak auth.

I am using shinyproxy-2.3.0.jar in a dockerized environment onto an Ubuntu 18.04.2 LTS machine. Nginx is used as reverse proxy in combination with a letsencrypt ssl cert.

Example-shiny-App:

library(shiny)
library(DT)

ui <- fluidPage(DT::dataTableOutput('tbl'))
server <- function(input, output, session) {
  output$tbl <- DT::renderDataTable(data.frame(Col_1=c(1:10), Col_2=c(11:20)))
}
shinyApp(ui, server)

Trying to run upper example with shinyproxy and keycloak-auth (btw. running on the same server) leads on to the mentioned 503 Service Unavailable AJAX error. Turning server=FALSE solves that error but is out of question for big tables.

Exact the same machine (docker,shinyproxy,nginx setting) with simple auth or ldap in application.yml is running without any trouble.

chienyuchuan commented 5 years ago

the same problem with billy34 in ShinyProxy 2.3.0, use DT::renderDataTable still have same ajax error 503 use keycloak v7.0.0 for Authentication/Authorization but reverse to 2.1 will work fine

Example App

library(shiny) ui <- fluidPage(DT::dataTableOutput('tbl')) server <- function(input, output, session) { output$tbl <- DT::renderDataTable({ DT::datatable(iris, class = 'cell-border stripe') }) }

vpanfilov commented 5 years ago

I have the same TruncatedResponseException mentioned by @bogdanrau with uploading files to RStudio running inside dockerized Shiny Proxy > 2.1 (using simple auth method). https://github.com/openanalytics/shinyproxy/issues/170

dpavlic commented 4 years ago

Just adding a 'me too'. The issue is with Keycloak authorization backend and does not otherwise show up. Reverting to 2.1 works as expected.

tjpalanca commented 4 years ago

Encountering this also with KeyCloak Auth and any ShinyProxy version > 2.1.0

jaysnm commented 4 years ago

Encountered same problem in nginx (ssl enabled) and keycloak auth service. However, DT displays perfectly without keycloak. Please see attached!

Error-Traceback

Http-Response

jaysnm commented 4 years ago

Encountered same problem in nginx (ssl enabled) and keycloak auth service. However, DT displays perfectly without keycloak. Please see attached!

Error-Traceback

Http-Response

ShinyProxy 2.3.0 jar from shinyproxy.io

tverbeke commented 4 years ago

We were able to reproduce with Keycloak. A fix will be in the next release.

jaysnm commented 4 years ago

We were able to reproduce with Keycloak. A fix will be in the next release.

Thanks for reverting!

kunal15145 commented 4 years ago

Getting the same error, without any kind of authentication. When making a POST request of content-type: multipart/form-data. Any idea what is the difference between the two versions that this error is coming. Please suggest some solution in case reverting is not an option. Btw I am running flask behind shinyproxy Thanks.

jaysnm commented 4 years ago

@tverbeke could you kindly push a snapshot of the <dev> branch with this issue fixed? perhaps early tests may unearth something else that may need a fix before updating the <stable> branch. Thanks in advance.

tverbeke commented 4 years ago

@jaysnm the fix is present in the develop branch of containerproxy.

jaysnm commented 4 years ago

@tverbeke it's unfortunate that develop branch of containerproxy build failed (please see attached). I have also built develop branch of shinyproxy hoping the containerproxy 0.8.4-SNAPSHOT on pom.xml would give the purported fix. No fix yet. At this point I rest my case for this is something else different from what is tracked by this issue. Thanks.

image

tverbeke commented 4 years ago

@jaysnm You need to compile with Oracle Java for the moment. The constructor in the sun.security.krb5.KrbTgsReq class is different between Oracle Java and OpenJDK and we are aware of it.

Dusan-Dingarac commented 4 years ago

With official version 2.3.1 this works finally fine for me. Time to close this issue #140?

tverbeke commented 4 years ago

Thanks for the feedback, @Dusan-Dingarac

giordyb commented 4 years ago

Hi, I'm having the same issue after upgrading from ShinyProxy 2.3.1 to 2.4.1, can we re-open the bug?

on browser: DataTables warning: table id=DataTables_Table_0 - Ajax error. For more information about this error, please see http://datatables.net/tn/7

on shinyproxy at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:548) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at eu.openanalytics.containerproxy.util.ProxyMappingManager$1.getConnection(ProxyMappingManager.java:88) ~[containerproxy-0.8.5.jar!/:0.8.5] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:310) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ 2020-10-23 15:33:30.211 DEBUG 1 --- [ XNIO-1 I/O-2] io.undertow.request.error-response : Setting error code 503 for exchange HttpServerExchange{ POST /system-recommendation-dashboard/pr │ │ oxy_endpoint/3913e3ea-8e4d-4c7a-9bf7-4418b3064b26/session/2baa425f53c271bc590b7d1a0c335a89/dataobj/tab_Cov_Linea_O} │ │ java.lang.RuntimeException: null │ │ at io.undertow.server.HttpServerExchange.setStatusCode(HttpServerExchange.java:1416) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler.handleFailure(ProxyHandler.java:668) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$HTTPTrailerChannelListener.handleEvent(ProxyHandler.java:769) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:646) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientExchange.invokeReadReadyCallback(HttpClientExchange.java:212) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.initiateRequest(HttpClientConnection.java:414) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.sendRequest(HttpClientConnection.java:347) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction.run(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.util.SameThreadExecutor.execute(SameThreadExecutor.java:35) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.HttpServerExchange.dispatch(HttpServerExchange.java:821) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:290) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connectionReady(ProxyConnectionPool.java:353) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:548) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at eu.openanalytics.containerproxy.util.ProxyMappingManager$1.getConnection(ProxyMappingManager.java:88) ~[containerproxy-0.8.5.jar!/:0.8.5] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:310) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final]

billy34 commented 4 years ago

I'm having the same issue with DT without ShinyProxy with load balancing using traefik as reverse proxy. Maybe linked to DT and load balancing ? Need to investigate on my side Have you seen this issue https://github.com/rstudio/DT/issues/849 ?

giordyb commented 4 years ago

I'm also using traefik as a load balancer but I went back to 2.3.1 and it works fine, with 2.4.0 and 2.4.1 it gives that error.

LEDfan commented 4 years ago

Hi @giordyb

I just tested the example code in the first post of this issue with ShinyProxy 2.4.1 and that still works for me. Can you share the following information:

I need at least the two first pieces, otherwise I won't be able to re-produce the issue. Thanks

shrektan commented 4 years ago

Just add my two cents here. In order to ensure DT works behind a load balancer (Traefik) , I recently created a simple repo DT-load-balancer based on Docker swarm, with detailed setups... I think with a little effort, you can turn it into a reproducible example for ShinyProxy about this issue...

billy34 commented 4 years ago

@giordyb I solved my problem with traefik and load balancing. I'm still using traefik 1.7 (1.7.26) and there is still a problem with stickiness configuration. I was using label "traefik.backend.loadbalancer.stickiness" as documented but with no success. Using deprecated label "traefik.backend.loadbalancer.sticky=true" is working. Don't know if it's a clue for this issue but who knows.

giordyb commented 4 years ago

@billy34 thanks, I am running traefik 2.2, i tried to enable the sticky session but it didn't make any difference. I will post my the code and traefik config shortly

giordyb commented 4 years ago

@LEDfan I'm running traefik under kubernetes (on a multi-node cluster) so it's a bit tricky to replicate our environment with a docker-compose. I was able to create a smaller environment with a single kubernetes node under docker (docker for mac) and it still errors out.

here is the code for the app (it's just a sample but it errors out as well): edit: see the next post for the code

I tested this version of this app with this config of traefik under shinyproxy 2.3.0 and it works, if I try the same thing on 2.4.1 I get the error

Another thing that I tried is to bypass traefik by port-forwarding directly to the shinyproxy container, even by doing this the error shows up, so I really doubt that it's a traefik issue.

giordyb commented 4 years ago

@LEDfan

I did some more tests, it looks like traefik has nothing to do with it.

I created a repo that replicates my environment (https://github.com/giordyb/test_shinyproxy.git)

on docker for mac running kubernetes I created 2 deployments of shiny proxy (2.3.1 and 2.4.1) that serve the same shiny R container application. There is no traefik involved, the shiny proxy pods are accessed via a kube-proxy port-forward.

under 2.3.1 the request to the datatable works

under 2.4.1 the same request fails

and I get this in the log of the shinyproxy container 2020-10-26 15:17:46.359 ERROR 1 --- [ XNIO-1 I/O-2] io.undertow.proxy : UT005028: Proxy request to /proxy_endpoint/884afa30-73a0-444f-b04f-3ae85b6de6bd/session/aacc55466aefc00eabd04b31f9891242/dataobj/tab_Cov_Linea_O failed │ │ io.undertow.server.TruncatedResponseException: null │ │ at io.undertow.client.http.HttpRequestConduit.truncateWrites(HttpRequestConduit.java:711) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.terminateWrites(AbstractFixedLengthStreamSinkConduit.java:256) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at org.xnio.conduits.ConduitStreamSinkChannel.shutdownWrites(ConduitStreamSinkChannel.java:178) ~[xnio-api-3.8.0.Final.jar!/:3.8.0.Final] │ │ at io.undertow.channels.DetachableStreamSinkChannel.shutdownWrites(DetachableStreamSinkChannel.java:79) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$HTTPTrailerChannelListener.handleEvent(ProxyHandler.java:754) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:646) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientExchange.invokeReadReadyCallback(HttpClientExchange.java:212) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.initiateRequest(HttpClientConnection.java:414) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.sendRequest(HttpClientConnection.java:347) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction.run(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.util.SameThreadExecutor.execute(SameThreadExecutor.java:35) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.HttpServerExchange.dispatch(HttpServerExchange.java:821) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:290) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connectionReady(ProxyConnectionPool.java:353) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:548) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at eu.openanalytics.containerproxy.util.ProxyMappingManager$1.getConnection(ProxyMappingManager.java:88) ~[containerproxy-0.8.5.jar!/:0.8.5] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:310) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ 2020-10-26 15:17:46.361 DEBUG 1 --- [ XNIO-1 I/O-2] io.undertow.request.error-response : Setting error code 503 for exchange HttpServerExchange{ POST /proxy_endpoint/884afa30-73a0-444f-b04f-3ae85b6de6bd/session/aacc55466aefc00eabd04b31f9891242 │ │ java.lang.RuntimeException: null │ │ at io.undertow.server.HttpServerExchange.setStatusCode(HttpServerExchange.java:1416) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler.handleFailure(ProxyHandler.java:668) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$HTTPTrailerChannelListener.handleEvent(ProxyHandler.java:769) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:646) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction$1.completed(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientExchange.invokeReadReadyCallback(HttpClientExchange.java:212) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.initiateRequest(HttpClientConnection.java:414) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.client.http.HttpClientConnection.sendRequest(HttpClientConnection.java:347) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyAction.run(ProxyHandler.java:561) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.util.SameThreadExecutor.execute(SameThreadExecutor.java:35) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.HttpServerExchange.dispatch(HttpServerExchange.java:821) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.completed(ProxyHandler.java:290) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connectionReady(ProxyConnectionPool.java:353) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:548) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:316) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at eu.openanalytics.containerproxy.util.ProxyMappingManager$1.getConnection(ProxyMappingManager.java:88) ~[containerproxy-0.8.5.jar!/:0.8.5] │ │ at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:310) ~[undertow-core-2.1.4.Final.jar!/:2.1.4.Final] │ │ at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final] │ │ at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.0.Final.jar!/:3.8.0.Final]

is there anything else that I can do on my end to troubleshoot the issue further?

LEDfan commented 4 years ago

Hi @giordyb

Thank you for your example deployment and report. I looked into the issue today and it's quite a strange issue. The problem is caused due to having debug logging enabled. There is one class DispatcherServlet which will log the parts of the requests send through ShinyProxy. However, this causes issue with some proxied requests, because part of those requests can only be read once. Therefore the Shiny app received an invalid request.

The solution is to disable debug logging for the org.springframework.web.servlet.DispatcherServlet class as follows:

logging:
  level:
     root: DEBUG
     org:
        springframework:
           web:
              servlet:
                 DispatcherServlet: INFO

This will be the default in the next ShinyProxy release, so that this workaround is no longer needed.

giordyb commented 4 years ago

Hi @LEDfan,

thank you for looking into this so promptly. I added the logging configuration as you suggested and now everything is working as expected, so I'm a happy camper.

thank you very much for your help!

smlehman commented 4 years ago

I'm having the same issue with kubernetes behind NGINX and Azure Active Directory. Data Tables display correctly with 2.3.1 but will not with 2.4.1. I've tried builds with JDK8 and with JDK11.

We've removed all logging config from application.yml and have set NGINX to run 'sticky' sessions, but no luck so far.

Have posted a simplified version of our application.yml and shinyproxy pod logs on the community support page. https://support.openanalytics.eu/t/datatables-ajax-error-with-shinyproxy-v-2-4-1/1763 .

LEDfan commented 3 years ago

Hi @smlehman , thank you for your extra information.

Could you please provide an example Shiny app (Dockerfile and R sources) using DataTables? I have one app, but it does not give any problems on a Kubernetes setup (using nginx too). Thanks!

smlehman commented 3 years ago

Hi @LEDfan Sure thing. To give you the full picture, I've created a small repo with all the basic files we are using. I've also included the README that I'm working on to support folks interested in using the application, as it might provide additional context about our setup.

https://github.com/smlehman/shinyproxy-debugDT Let me know if there's anything else that would help with troubleshooting.

AkedoMakona commented 1 month ago

Hi @giordyb

Thank you for your example deployment and report. I looked into the issue today and it's quite a strange issue. The problem is caused due to having debug logging enabled. There is one class DispatcherServlet which will log the parts of the requests send through ShinyProxy. However, this causes issue with some proxied requests, because part of those requests can only be read once. Therefore the Shiny app received an invalid request.

The solution is to disable debug logging for the org.springframework.web.servlet.DispatcherServlet class as follows:

logging:
  level:
     root: DEBUG
     org:
        springframework:
           web:
              servlet:
                 DispatcherServlet: INFO

This will be the default in the next ShinyProxy release, so that this workaround is no longer needed.

This still seems to be necessary when using Shinyproxy-operator on AWS EKS with Shinyproxy-Monitoring enabled. I spent a half-day debugging strange un-logged crashes and this seems to have solved it.