Feature request: Have client send keep alive message to prevent TCP timeout on some load balancers

[shimberger]

We have deployed a shiny app behind an AWS Elastic Load Balancer. We have issues that the app disconnects after some time of inactivity because the load balancer disconnects the idle TCP connection. It would be nice to have a configuration option in shiny::runApp to have the client send a ping or keep-alive message every n seconds. This could just be a JSON message that get's send over the WebSocket in the form of

{
  method: 'ping'
  data: null,
}

or something like this. It doesn't sound hard to implement (I am familiar with JavaScript but not R) and would be willing to spend a few hours doing so if there is interest in that feature and I would be given some hints on where to start? E.g. add it in srcjs/shinyapp.js and somewhere in the R code.

Thanks.

If this is not interesting, should be solved otherwise etc. Please just tell me and close the ticket. I know there is always too much stuff to do ;)

[alandipert]

Can you please elaborate a little more on what you see during and after the disconnect?

I ask because if you're on an ELB, you must be using Shiny Server or Connect, since only those offer the ability to use non-WebSocket transports. And ELBs don't supports WebSockets.

So, I suspect what you're seeing is probably not related to WebSockets, but might be solved by other means, such as further configuration of Shiny Server/Connect/ELB.

One alternative to the configuration research route is to use an ALB instead of an ELB. They're like ELBs but they support WebSockets, and so don't require Shiny to fall back to any non-WebSocket transport. The one trick to them is that if you're running a cluster of Shiny Servers or Connects behind your ALB, you must configure sticky sessions.

[shimberger]

@alandipert Thanks so much for getting back to me so quickly.

You are indeed correct: I am using the Application Load Balancer (ALB). Sorry I wasn't precise. The symptom is that after the configured idle timeout (https://s3-eu-west-1.amazonaws.com/captured-krxvuizy1557lsmzs8mvzdj4/bp7p6-20180626-26091742.png) the screen goes gray.

This seems to be due to the fact that if the user is idle no traffic is flowing over the TCP connection of the WebSocket. I've reproduced this by measuring the time of when the screen greyes out and comparing it to the ALB timeout. I also changed the ALB timeout and the behaviour changed accordingly. I am not sure how sticky sessions could help but I am willing to give it a try if you still think that's what it is.

Just for the record: I am not using shiny server or connect. Just docker and shiny:runApp behind an Apache reverse proxy.

Thanks so much again.

[wch]

I've also seen this behavior on rstudio.cloud, which uses an ALB which is configured for a 5 minute timeout. If a user starts an app there and lets it sit without any interaction, the session closes after 5 minutes (although the app keeps running).

[dpmccabe]

Here's a simple solution to increment a counter every 10 seconds.

JS

var socket_timeout_interval;
var n = 0;

$(document).on('shiny:connected', function(event) {
  socket_timeout_interval = setInterval(function() {
    Shiny.onInputChange('alive_count', n++)
  }, 10000);
});

$(document).on('shiny:disconnected', function(event) {
  clearInterval(socket_timeout_interval)
});

CSS

#keep_alive {
  visibility: hidden;
}

server.R

output$keep_alive <- renderText({
  req(input$alive_count)
  input$alive_count
})

ui.R

textOutput("keep_alive")

[falaki]

We are encountering this issue as well. The idle timeout for the websocket connection in Shiny cannot be effectively controlled by Shiny unless it we implement a heartbeat mechanism.

Although there is an internal idle timeout that is said to be configurable in RStudio Connect, the websocket could still be terminated by other components along the path such as cloud load balancers, whichever has a shorter idle timeout. This is even more severe when not using RStudio Connect.

We encountered this issue with load balancers in AWS and Azure. The load balancer timeout could be as short as 1 minute, which could harm the user experience of shiny even for the development setting.

Shiny can consciously control or prevent the timeout end-to-end by introducing a heartbeat mechanism. See the websocket RFC section on ping/pong messages. With this mechanism, Shiny would be more robust in the cloud environments and open up opportunities for more use cases.

[ciaransweet]

@shimberger Did you ever get around this? I think I'm experiencing the same thing.

[iqis]

my solution, if I understood the problem correctly:

timer <- reactiveTimer(1000 * 60 * 5) # time unit in milliseconds
        observe({
            timer()
            {{"here insert your code to submit a message through websocket client"}}
        })

[tom-puckett]

A real option to use websocket Ping/Pong would be very helpful. My need is for deployment behind a proxy built with Microsoft YAPR library (100 second timeout), but I also expect to deploy in Azure where all idle connections are killed after 90 seconds. Ping is better than an application level message to keep log files clean. Please consider this.

https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers search for Pings and Pongs

[HugoGit39]

Currently trying all these timer solutions on Digital Ocean App and Droplet platforms without any luck....anyone know how to tackle this?

[gadenbuie]

As of httpuv 1.6.10, httpuv now sends ping frames to the websocket connection every 20 seconds. That may resolve some of the issues described in this thread.

@HugoGit39 which version of httpuv are you using?

[HugoGit39]

Hi @gadenbuie I havent used httpuv yet! Only @dpmccabe mentioned solution and also the other timer auto invalidate solutions....

Just checked it quickly....where do I have to add this to my Shiny Code?

[dpmccabe]

This issue was submitted 6 years ago. As far as I know, Shiny still stores state in process memory. Different web frameworks store state in a lot of different ways (browser localStorage/IndexedDB, server-side files, relational databases, key-value stores, etc.). There are good reasons why most frameworks don't store state solely in process memory. The fact that Shiny does is one of many reasons why I eventually concluded it's completely inappropriate for a production environment where you're forced to use hacky keepalive and sticky session techniques to get it working with a load balancer.

[gadenbuie]

Just checked it quickly....where do I have to add this to my Shiny Code?

@HugoGit39 Oh, sorry, I should have clarified: you don't have to use httpuv explicitly in your Shiny code. httpuv is used behind-the-scenes by Shiny to server Shiny apps. So you could just check packageVersion("httpuv").

[HugoGit39]

Just checked it quickly....where do I have to add this to my Shiny Code?

@HugoGit39 Oh, sorry, I should have clarified: you don't have to use httpuv explicitly in your Shiny code. httpuv is used behind-the-scenes by Shiny to server Shiny apps. So you could just check packageVersion("httpuv").

I have it running via a Docker file using r2u:22.04 at Digital Ocean....how do I check the version there?

[gadenbuie]

I have it running via a Docker file using r2u:22.04 at Digital Ocean....how do I check the version there?

@HugoGit39 I think it's safe to assume that you're using a recent-enough version of httpuv. I'm sorry I can't help much with the specifics of Digital Ocean. Are you following a tutorial or guide? Is your code publicly available?

[HugoGit39]

@gadenbuie no sorry my code is not available.

I also tried Heroku, however with every 'heart-beat' method the app still greys out..

I dont get it cause @virtualstaticvoid posted a solution:

https://github.com/virtualstaticvoid/heroku-buildpack-r/issues/126

which he tried out on Heroku 2 weeks ago...but yet doesnt work for me

[HugoGit39]

Still trying..also with Azure...no luck

tbh I dont understand why this 6 year issue is still open and has a low prio...even a hacky solution would be great

I mean I spent literally 1000+ hours building my app, learning Shiny, docker etc...and than realizing the the final final final step cant be resolved...you cant be serious

[gadenbuie]

@HugoGit39 I certainly understand the frustration of having invested time in developing your app and struggling to deploy it. However, the specific request in this issue has been solved a year ago, see rstudio/httpuv#291. Also, as I'm sure you're aware, Posit provides a mix of free and paid services for deploying Shiny apps. These services support continued development of Shiny and aim to make app deployment a less frustrating experience.

A difficulty that you and others in this thread face is that custom deployment environments are by definition unique. Without detailed steps to recreate your environment with an app that reproduces your issue, it's difficult for anyone to provide advice unless they've happened to encounter the exact same problem in their own work.

[jcheng5]

@HugoGit39 We should maybe back up a bit—is there a reason you suspect that the grey outs you’re experiencing are due to this specific issue of WebSocket timeout, and not a different reason? The screen greys out whenever the WebSocket is disconnected, and there are lots of reasons that can happen. One of the most common is an observe() in the app throws an unhandled exception. Have you looked at the stderr output of the R process when the grey out happens? How about the output of the JavaScript console?

Edit: Another reason might be lack of RAM. If your server runs out of memory is not configured with swap (or runs out of swap) the Linux “oom-killer” process will kill app and you won’t see anything in the logs except maybe the word “Killed”.

[HugoGit39]

Hi thx for all your help..and @gadenbuie indeed I am aware of the less obtrusive solutions...but they are pretty expensive, which is not really workable if you do this as a hobby

@jcheng5 a nice websocket-test example has been created today by @virtualstaticvoid

https://github.com/virtualstaticvoid/heroku-docker-r-websocket-test

This is a basic R Shiny app with a heart-beat mechanism, whihc works with @virtualstaticvoid system.....I tested it with Chrome, Edge and Firefox but still timeout. So somehow my web-browser settings are different?