"Execution halted" on shiny app that just finished loading, causing grey out

[chrisirhc]

I''m attempting to debug a shiny application and it looks like the log files are being deleted right after "Execution halted", though I don't know why the execution halted. I saw #82 where the logs are being deleted by exit code.

I'm not sure how but I guess "Execution halted" can still result in a non-error exit code?

[jcheng5]

"Execution halted" is the normal message printed when the Shiny process is gracefully shut down by Shiny Server. Can you tell us more about what's going wrong that you're trying to debug in the first place? On Mon, Nov 9, 2015 at 6:46 PM Chris Chua notifications@github.com wrote:

I''m attempting to debug a shiny application and it looks like the log files are being deleted right after "Execution halted", though I don't know why the execution halted. I saw #82 https://github.com/rstudio/shiny-server/issues/82 where the logs are being deleted by exit code.

I'm not sure how but I guess "Execution halted" can still result in a non-error exit code?

— Reply to this email directly or view it on GitHub https://github.com/rstudio/shiny-server/issues/155.

[chrisirhc]

I have a Shiny app where I load and some times takes a while to load (~20 seconds), but when it's done loading it ends up greyed out. Hence, I'm investigating whether the application is somehow crashing after loading. This doesn't happen whenever it loads quickly (which sometimes happens randomly).

Does the shiny-server have some idle timeout perhaps that it might be hitting even though it's still initializing?

[jcheng5]

There is an app_init_timeout config value that you can add to shiny-server.conf: http://rstudio.github.io/shiny-server/latest/#app_init_timeout

Maybe try setting that to 120 in /etc/shiny-server/shiny-server.conf, then reload or restart Shiny Server. On Mon, Nov 9, 2015 at 6:54 PM Chris Chua notifications@github.com wrote:

I have a Shiny app where I load and some times takes a while to load (~20 seconds), but when it's done loading it ends up greyed out. Hence, I'm investigating whether the application is somehow crashing after loading. This doesn't happen whenever it loads quickly.

Does the shiny-server have some idle timeout perhaps that it might be hitting even though it's still initializing?

— Reply to this email directly or view it on GitHub https://github.com/rstudio/shiny-server/issues/155#issuecomment-155267131 .

[chrisirhc]

Hm okay, would be helpful to add a default to the documentation. I'll try that and see if it works. Thanks!

[jcheng5]

Yeah, sorry, I was really surprised I did not document the default value. On Mon, Nov 9, 2015 at 7:05 PM Chris Chua notifications@github.com wrote:

Hm okay, would be helpful to add a default to the documentation. I'll try that and see if it works. Thanks!

— Reply to this email directly or view it on GitHub https://github.com/rstudio/shiny-server/issues/155#issuecomment-155269380 .

[chrisirhc]

No problem!

I just tried it out, it doesn't seem to fix the issue I'm having. Somehow the application still seems to quit, right after loading.

[jcheng5]

Any error in the browser's JavaScript console? On Mon, Nov 9, 2015 at 8:00 PM Chris Chua notifications@github.com wrote:

No problem!

I just tried it out, it doesn't seem to fix the issue I'm having. Somehow the application still seems to quit, right after loading.

— Reply to this email directly or view it on GitHub https://github.com/rstudio/shiny-server/issues/155#issuecomment-155277219 .

[chrisirhc]

Nope, just seeing:

Connection closed. Info: {"type":"close","code":1006,"reason":"Server lost session","wasClean":false}

[chrisirhc]

This only happens when not using websockets, due to a proxy in the middle, perhaps there's some connection timeout in the xhr polling.

[jcheng5]

That would make a lot of sense. I think our config sends traffic every 25 seconds to prevent timeouts--is it possible your proxy is more aggressive in timing out than that? I think if that's the case, even if an app works, just leaving it idle for 25 seconds should grey it out--is that the case? On Thu, Nov 12, 2015 at 7:45 PM Chris Chua notifications@github.com wrote:

This only happens when not using websockets, due to a proxy in the middle, perhaps there's some connection timeout in the xhr polling.

— Reply to this email directly or view it on GitHub https://github.com/rstudio/shiny-server/issues/155#issuecomment-156314978 .

[chrisirhc]

I looked into this possibility. However, it still looks like something is up with shiny-server or SockJS. I put a clean nginx configuration just without the websocket upgrade headers (to replicate the fallback to xhr-polling). However, I still see the following in the Chrome DevTools (and the shiny app is greyed out):

It looks like the last request to the xhr endpoint returned a blank response and was less than 1 second and SockJS takes this as the server doing a disconnect (see console log in the above screenshot).

Here's a screenshot of connecting through the same nginx reverse proxy but with WebSocket upgrade headers enabled in the configuration and nothing else changed (app is working as per normal, not greying out):

[Mkamar]

I am having the same issue, any luck finding out why this happens?

[benb111]

"It looks like the last request to the xhr endpoint returned a blank response and was less than 1 second and SockJS takes this as the server doing a disconnect (see console log in the above screenshot)."

I am having this same issue. It happens sporadically, usually 3-60 minutes after I open my app and let it sit idly. It goes away when I connect directly with websockets, but that may not be an option for us with our proxy setup.

[zippeurfou]

+1 same issue here. Seems to be caused on the Shiny Server v1.4.1.759 (ubuntu). However, when I try on the Shiny Server v1.4.1.0 (mac os x). It works.

[vnniyB]

I'm getting the same issue, application takes ~10 seconds to load, displays valid output for ~5 seconds, then greys out.

here's the logs from /var/log/shiny-server.log

ESC[34m[2016-01-14 21:34:06.691] [TRACE] shiny-server - ESC[39mProxying request: /AdminLTE-2.0.6/app.min.js.map
ESC[34m[2016-01-14 21:34:06.704] [TRACE] shiny-server - ESC[39mWorker #5fd50a54744c8b32 releasing http port. 0 open HTTP connection(s), 0 open WebSocket connection(s).
ESC[34m[2016-01-14 21:34:06.704] [TRACE] shiny-server - ESC[39mNo clients connected to worker #5fd50a54744c8b32. Starting timer
ESC[34m[2016-01-14 21:34:07.798] [TRACE] shiny-server - ESC[39mAccepted SockJS connection for /BlockPlans/__sockjs__/527/tzi8rtjx/xhr_streaming
ESC[34m[2016-01-14 21:34:07.804] [TRACE] shiny-server - ESC[39mReusing existing instance
ESC[34m[2016-01-14 21:34:07.805] [TRACE] shiny-server - ESC[39mWorker #5fd50a54744c8b32 acquiring sock port. 0 open HTTP connection(s), 1 open WebSocket connection(s).
ESC[34m[2016-01-14 21:34:18.923] [TRACE] shiny-server - ESC[39mWorker #5fd50a54744c8b32 releasing sock port. 0 open HTTP connection(s), 0 open WebSocket connection(s).
ESC[34m[2016-01-14 21:34:18.924] [TRACE] shiny-server - ESC[39mNo clients connected to worker #5fd50a54744c8b32. Starting timer
ESC[34m[2016-01-14 21:34:23.926] [TRACE] shiny-server - ESC[39mTimeout expired. Killing process.
ESC[34m[2016-01-14 21:34:23.928] [TRACE] shiny-server - ESC[39mInterrupting process on socket port 40627
ESC[34m[2016-01-14 21:34:23.928] [TRACE] shiny-server - ESC[39mSending SIGINT to 33340
ESC[34m[2016-01-14 21:34:25.063] [TRACE] shiny-server - ESC[39mPort 40627 returned
ESC[34m[2016-01-14 21:34:25.064] [TRACE] shiny-server - ESC[39mPort 40627 proxy closed
ESC[34m[2016-01-14 21:34:25.065] [TRACE] shiny-server - ESC[39mNormal exit, deleting log file ...

Shiny-server version: 1.4.0.721 Shiny package version: 0.12.1 R version: 3.2.0

[jaearick]

Having the same problem with shiny-server version 1.4.2.786 and R 3.2.3-4 (rpms). The same code works on an earlier server running shiny 1.3.0.403 and R 3.2.3-4. I put both servers into trace mode. The earlier server uses websocket for SockJS:

shiny-server - Accepted SockJS connection for ...../websocket

I cannot get the new server to use websocket no way, no how. It uses items like xhr, eventsource, etc but no websockets. In my /etc/shiny-server/shiny-server.conf file I added at the top:

app_init_timeout 1800; app_idle_timeout 1800; disable_websockets off; disable_protocols xdr-streaming xhr-streaming iframe-eventsource iframe-htmlfile xdr-polling xhr-polling iframe-xhr-polling jsonp-polling;

Basically I try to force websockets by turning everything else off. Still does not work, still greyed out. I think this is a bug in SockJSAdapter.R.

[jcheng5]

@jaearick @vnniyB @zippeurfou @benb111 @Mkamar Are you all running shiny-server behind an nginx proxy?

[jaearick]

In my case, both servers run apache 2.2.31 with the following proxy modules in place and loaded via httpd.conf:

LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_scgi_module modules/mod_proxy_scgi.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so LoadModule proxy_html_module modules/mod_proxy_html.so

The httpd.conf has the following for proxy settings on both servers:

--- Shiny Server proxy

webservices proxy must be before the generic proxy

ProxyPassMatch ^/(./sockjs/[0-9]+/.)$ ws://127.0.0.1:3838/$1

generic proxy

ProxyRequests Off ProxyPass / http://localhost:3838/ ProxyPassReverse / http://localhost:3838/

RewriteRule ^/(.*)$ http://localhost:3838/$1 [P]

SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1

I've looked high and low for config differences between the two servers. With the exception of a few rpm differences and version differences in shiny and some shiny packages installed, they are the same. The old server had poorly laid out disk. The new version is a revamp. Both are Centos 7.2.

[jaearick]

Well, a big "thank you" to Joe Cheng for continually bringing up nginx. This (and a good night's sleep) convinced me to go back and carefully read:

https://support.rstudio.com/hc/en-us/articles/213733868-Running-Shiny-Server-with-a-Proxy

In my case, the bottom half regarding Apache. Once I changed the entry in httpd.conf for ProxyPassMatch from:

ProxyPassMatch ^/(./sockjs/[0-9]+/.)$ ws://127.0.0.1:3838/$1 (worked on the old system) to ProxyPassMatch ^/(.+)/websocket ws://127.0.0.1:3838/$1/websocket (copied from webpage above)

then things started working correctly. The TRACE output of shiny-server.log showed websocket connections. Yippee!

[zippeurfou]

Yes for me I had to do something close to it:

<VirtualHost *:80>
  RewriteEngine On
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteRule /(.*) ws://localhost:3839/$1 [P,L]

  ProxyPass / http://localhost:3839/

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        ProxyPass http://localhost:3839/
        ProxyPassReverse  http://localhost:3839/
        AuthType Basic
        AuthName "Restricted Access - Authenticate"
        AuthUserFile /home/ubuntu/passwords
        Require valid-user
    </Location>

</VirtualHost>

[benb111]

@jcheng5 Yes I was behind nginx (and actually another Apache instance and an AWS ELB). Eventually once I got all three proxy layers supporting websockets, the problem went away. But as soon as I disabled websockets it returned. So it seems like there is some issue in the fallback to xhr polling in sockjs, which I would expect to work for users on older browsers or behind network layers that don't support websockets.

[rajeshpudota]

Facing the same issue. When the proxy is on, the application gets greyed out. On the contrary, when the proxy is off, the application works well.

I actually need my app to work when the proxy is on, as it has some other data dependencies on proxy.

Do we have any work around to make the application work even when the proxy is on ?

[kissmygritts]

My app is behind nginx and adding the following 3 lines to my /etc/nginx/sites-enabled/default file solved the websockets issues. I had similar behavior from the photo @chrisirhc posted.

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

[olcaysah]

@kissmygritts solution solved my same problem.

[kammerer]

In my case it was old nginx version which did not support proxy_http_version directive. Upgrade and adding proxy_http_version 1.1; fixed the issue.

[jjesusfilho]

I also use rstudio-server with Nginx and had the same problem, but solved after adding the three lines suggested by @kissmygritts. The difference is that I added them to /etc/nginx/conf.d/default.conf which is a more direct approach. I found this helpful: https://stackoverflow.com/a/25435118/5309223

[GitHunter0]

@jjesusfilho , @kissmygritts , I'm running both nginx and shiny server as docker containers via docker-compose on Ubuntu. How do I correct it in this case, is it enough to alter the nginx.conf file? I appreciate if you can help.

[sanjmeh]

Looking at nine thumbs ups I am also copying these in my nginx conf.

My app is behind nginx and adding the following 3 lines to my /etc/nginx/sites-enabled/default file solved the websockets issues. I had similar behavior from the photo @chrisirhc posted.

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";