Shiny doesn't work when deployed on Google Cloud Run

[jdwrink]

Hello,

I am attempting to deploy a Shiny app to Google Cloud Run. I am getting http 400 errors when my browser tries to download static files, such as jquery and bootstrap related files. This renders the app unusable. I think this behavior may be related to this bug. However, the suggested fix, upgrading to Shiny 1.3.2 and using the development version of httpuv, isn't working. Also, since this is a Google managed service, it is not possible to alter the proxy settings.

System details

Output of sessionInfo():

[1] "R version 3.6.0 (2019-04-26)"                                                
 [2] "Platform: x86_64-pc-linux-gnu (64-bit)"                                      
 [3] "Running under: Debian GNU/Linux 9 (stretch)"                                 
 [4] ""                                                                            
 [5] "Matrix products: default"                                                    
 [6] "BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so"                               
 [7] ""                                                                            
 [8] "locale:"                                                                     
 [9] " [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              "                  
[10] " [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    "                  
[11] " [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             "                  
[12] " [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 "                  
[13] " [9] LC_ADDRESS=C               LC_TELEPHONE=C            "                  
[14] "[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       "                  
[15] ""                                                                            
[16] "attached base packages:"                                                     
[17] "[1] stats     graphics  grDevices utils     datasets  methods   base     "   
[18] ""                                                                            
[19] "other attached packages:"                                                    
[20] "[1] shiny_1.3.2"                                                             
[21] ""                                                                            
[22] "loaded via a namespace (and not attached):"                                  
[23] " [1] compiler_3.6.0    magrittr_1.5      R6_2.4.0          promises_1.0.1   "
[24] " [5] later_0.8.0       tools_3.6.0       htmltools_0.3.6   Rcpp_1.0.1       "
[25] " [9] jsonlite_1.6      digest_0.6.19     xtable_1.8-4      httpuv_1.5.1.9000"
[26] "[13] mime_0.6         "                                                      

Example dockerfile for deploying to Google Cloud Run

FROM rocker/r-ver

RUN groupadd shiny && useradd -r -m shiny -g shiny

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
     pandoc \
     pandoc-citeproc \
     libcurl4-gnutls-dev \
     libcairo2-dev \
     libxt-dev \
     xtail \
     libssl-dev \
     libicu-dev \
     wget

RUN R -e "install.packages(c('remotes', 'shiny', 'rmarkdown'), repos = 'https://cran.rstudio.com/', method='wget')"

RUN R -e "remotes::install_github(c('rstudio/httpuv'))"

RUN mkdir -p /srv/shiny && \
    chown shiny:shiny /srv/shiny

COPY old_faithful.R /srv/shiny/app.R

EXPOSE 8080

USER shiny

CMD ["R", "-e", "shiny::runApp('/srv/shiny', host='0.0.0.0', port=8080, launch.browser = F)"]

Example Shiny app, using the Old Faithful dataset. I altered the code to display the session info.

library(shiny)

ui <- bootstrapPage(

  selectInput(inputId = "n_breaks",
              label = "Number of bins in histogram (approximate):",
              choices = c(10, 20, 35, 50),
              selected = 20),

  checkboxInput(inputId = "individual_obs",
                label = strong("Show individual observations"),
                value = FALSE),

  checkboxInput(inputId = "density",
                label = strong("Show density estimate"),
                value = FALSE),

  plotOutput(outputId = "main_plot", height = "300px"),

  verbatimTextOutput("sessionInfo"),

  # Display this only if the density is shown
  conditionalPanel(condition = "input.density == true",
                   sliderInput(inputId = "bw_adjust",
                               label = "Bandwidth adjustment:",
                               min = 0.2, max = 2, value = 1, step = 0.2)
  )

)

server <- function(input, output) {

  output$sessionInfo <- renderPrint({
    capture.output(sessionInfo())
  })

  output$main_plot <- renderPlot({

    hist(faithful$eruptions,
         probability = TRUE,
         breaks = as.numeric(input$n_breaks),
         xlab = "Duration (minutes)",
         main = "Geyser eruption duration")

    if (input$individual_obs) {
      rug(faithful$eruptions)
    }

    if (input$density) {
      dens <- density(faithful$eruptions,
                      adjust = input$bw_adjust)
      lines(dens, col = "blue")
    }

  })
}

shinyApp(ui = ui, server = server)

To replicate

1. Build the docker file and deploy it to Google Container Registry.
2. Run this command to deploy the app: gcloud beta run deploy shiny-cloudrun-test --image gcr.io/[PROJECT-ID]/[IMAGE-NAME] --region us-central1
3. After the app is deployed, a URL is automatically generated. Follow the link.

Expected result:

Actual result:

[alandipert]

Hi, thank you for the report. Did this previously work for you with Shiny 1.2.0?

[alandipert]

Note: I wasn't able to reproduce this locally using the provided Dockerfile and app. I suspect something particular to Google Cloud Run must be involved, like the URL or the way it conveys headers to and from the Shiny app.

[jdwrink]

@alandipert After a day, I did figure out a workaround. I had to downgrade to Shiny 1.2.0, and use Shiny Server. I found that the only protocol that works on GCRun is json-polling. Any other protocol grays out the screen.

I also tried using Shiny 1.3.2, with the dev version of httpuv, and with json-polling. I got the same 400 bad request bug with static files.

I rebuilt the above code and ran it locally with this command:

PORT=8080 && docker run -p 8080:${PORT} -e PORT=${PORT} [IMAGE-NAME]

It ran for me. I was able to use the app at localhost:8080. I am using macOS Mojave and Docker for Mac 18.09.1. Was there a particular error you got from docker?

[alandipert]

@jdwrink thanks for the additional details. Sorry I wasn't clear; by "wasn't able to reproduce" I meant that the Dockerfile you provided worked for me locally.

It sounds like there might be two different issues here; one related to WebSocket (maybe under certain circumstances the GCR load balancer/reverse proxy isn't relaying WebSockets, necessitating Shiny Server?) and the other related to static files in 1.3.2. Does that sound plausible based on what you've observed?

I have no experience with GCR, but could it be there are settings you can change related to WebSockets? I wonder if support for them needs to be turned on explicitly. Another thing you might need to turn on is "stickiness"; that's a property of a load balancer that ensures requests from the same client are all routed to the same server.

Are you an RStudio customer using Shiny Server Pro? If so, you might consider contacting support@rstudio.com; they might be able to provide additional help.

Anyway, thanks again for the report and the additional information.

[jdwrink]

@alandipert I found an unofficial FAQ maintained by a Googler. Apparently websockets are not supported on Google Cloud Run at this time. They do work on Google Cloud Run for GKE though, and I know from personal experience that Shiny has no issues with Knative (the open source project GCR is based on).

So, the websocket issue and the static files issue are separate. Websockets are not a problem as long as users know to set up shiny server config to use json-polling. I plan to create a repo on my own Github account this weekend with a hello world example.

I am not a Shiny Server Pro customer, just a civilian. But I appreciate the product and the work RStudio does for our community.

[alandipert]

Thanks for digging into it. It sounds like we might be looking at an interaction between static files in Shiny 1.3.2 and Shiny Server. This is definitely something we'll look into.

[jcheng5]

@jdwrink It makes sense that websocket SockJS would fail, but I'm slightly surprised that xhr-streaming fails, and quite surprised that xhr-polling fails too. Maybe something to do with CORS or something???

[jcheng5]

Wait, is GCR akin to Amazon's Lambda? If so, I imagine this won't be a good fit for Shiny, no matter what software you put in the middle. These services are designed for stateless HTTP servers, and Shiny is inherently stateful. I bet you'd end up with errors under load as requests that can only be served out of container A (where its session lives in memory) end up being routed to container B instead.

[jdwrink]

@jcheng5 I was mistaken, xhr-polling does work. As to whether or not Cloud Run is suitable, it probably wouldn't be for a scenario where you would host an app for the masses (like on shinyapps.io). I believe it would be appropriate if you intended for the app to only serve a very small number of people, so few that it wouldn't scale beyond one running container.

[michaeleekk]

I was trying to deploy Shiny Application to Cloud Run yesterday with the latest version of Shiny 1.3.2 and found exactly the same issue, 400 aborted on static assets.

And the workaround proposed to downgrade to 1.2.0 and using xhr-polling works ! I disabled all protocols but left only xhr-polling and jsonp-polling with,

disable_protocols websocket xdr-streaming xhr-streaming iframe-eventsource iframe-htmlfile xdr-polling iframe-xhr-polling;

I wish the issue could be resolved, but, in the meantime, I guess I will stick with the workaround approach.

Thanks @jdwrink for opening the issue :)

[sylus]

Also running into this with an App running behind istio as an ingress gateway. Though would be an issue with any ingress.

Is a fix being worked on for this? Thanks for the awesome product!

[sylus]

Updating the httpuv package from rstudio fixed the problem for me :D thx all!

[slink42]

Tested working using google cloud run managed service with:

• shiny version 1.4.0
• all protocols disabled except xhr-polling and jsonp-polling
• rstudio/httpuv installed

My dockerfile and config will be up in the next day or so here: https://github.com/Artificially-Intelligent/shiny/releases as release 0.2

[jdwrink]

I can confirm what @slink42 posted. I was able to get the Old Faithful shiny app to work on Cloud Run with shiny 1.4.0 and the development version of httpuv (1.5.2.9000), with all protocols disabled except xhr-polling and jsonp-polling.

Should I close this issue, or wait until the working version of httpuv is on CRAN?

[wch]

I'm OK with keeping this open until httpuv is on CRAN.

For reference, this is the PR that fixed the issue (I believe): https://github.com/rstudio/httpuv/pull/248

Installing the dev version of httpuv should fix the issue:

# Install with remotes
remotes::install_github('httpuv')
# Or, with devtools
devtools::install_github('httpuv')

[cphthomas]

Hi

Im new to docker, I've had some difficulties disabling protocols same grey screen as in the shiny app screen @jdwrink had.

You can se my mini sample in cloud run here: https://console.cloud.google.com/home/dashboard?project=projm-268909

I am trying to run github repos from @jdwrink: Google Cloud Run and First fully functioning version from here: https://github.com/Artificially-Intelligent/shiny/releases

I get this error in terminal when I try your First fully functioning version (Mac OSX 10.15.3):

Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2019-12-12/src/contrib:
  cannot open URL 'https://mran.microsoft.com/snapshot/2019-12-12/src/contrib/PACKAGES'
Error in library(readr) : there is no package called ‘readr’
Calls: source -> withVisible -> eval -> eval -> library
In addition: Warning message:
package ‘readr’ is not available (for R version 3.6.1) 
Execution halted

Does you have a minimal example that works for GCR? I myself have this mini example but I cannot disable websockets :O)

I have tried to add this line to the Dockerfile:

COPY shiny-customized.config /etc/shiny-server/shiny-server.conf

Where the content of the shiny-customized.config is (I have tried many different things :O) )

disable_protocols websocket xdr-streaming xhr-streaming iframe-eventsource iframe-htmlfile xdr-polling iframe-xhr-polling;

# Instruct Shiny Server to run applications as the user "shiny"
run_as shiny;

# Define a server that listens on port defined by ENV variable PORT, defaults to 3838
server {
  listen ${PORT};

  # Define a location at the base URL
  location / {

    # Host the directory of Shiny Apps stored in this directory
    site_dir /02_code;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;
  }
}

Any help especially a small working sample would be greatly appreciated :O)

betty2.zip

[randy3k]

For future readers, it is a minimal example for the setup described by @michaeleekk and @slink42 https://github.com/randy3k/shiny-cloudrun-demo

[MarkEdmondson1234]

Yes Shiny on Cloud Run is working but you need to limit it to one container (not the default 1000), however that container can accept up to 80 connections at once.

So its not quite "scale to a billion" as you may want, but it does scale to 0 (no cost when no connections) and you will need to keep an eye on what compute load the Shiny app does to see if 80 concurrent connections are too much, but I guess that should be sufficient for a lot of use cases - you can also play with CPU and Memory settings. Thanks @randy3k for the demo app above that also disables the right websocket functions.

You can see a working version here https://shiny-cloudrun-ewjogewawq-ew.a.run.app/ that was deployed via googleCloudRunner code running my fork:

library(googleCloudRunner)

repo <- cr_buildtrigger_repo("MarkEdmondson1234/shiny-cloudrun-demo")
cr_deploy_docker_trigger(
  repo,
  image = "shiny-cloudrun"
)

cr_run(sprintf("gcr.io/%s/shiny-cloudrun:latest",cr_project_get()),
       name = "shiny-cloudrun",
       concurrency = 80,
       max_instances = 1)

[algo-se]

Wait, is GCR akin to Amazon's Lambda? If so, I imagine this won't be a good fit for Shiny, no matter what software you put in the middle. These services are designed for stateless HTTP servers, and Shiny is inherently stateful. I bet you'd end up with errors under load as requests that can only be served out of container A (where its session lives in memory) end up being routed to container B instead.

@jcheng5 What if you put the session data in Redis, via memory store?

[MarkEdmondson1234]

I don't think you need worry about that now, Cloud Run these days has support for stateful applications via session affinity https://cloud.google.com/run/docs/configuring/session-affinity

[algo-se]

Hey @MarkEdmondson1234 thank you for the answer. So with session affinity can you have n containers and "scale to a billion"?

[nick-youngblut]

Should I close this issue, or wait until the working version of httpuv is on CRAN?

I'm also getting a lot of http 400 errors when my browser tries to download static files, such as jquery and bootstrap related files.

I'm using httpuv 1.6.11 and shiny 1.7.5.

The work-around specified at https://github.com/randy3k/shiny-cloudrun-demo/tree/master might be helping (Thanks @randy3k !), but it at least didn't completely solve the issue.

I'm using a load balancer, custom domain, and IAP.

Update: setting --max-instances=1 seems to have fixed the issue.