Sockets suddenly stop recieving new messages (howewer it cloud works in another tab at the same time)

[chekalsky]

Sometimes websockets in different browsers just stops to receiving the new messages. Browser doesn't disconnects or something, it just don't see new messages. Howewer in another tab it could work fine. There is no difference on how long server running or how old is tab, problem seems to shown up randomly.

Nginx restart helps for a while.

stat:

<channels>8024</channels>
<wildcard_channels>0</wildcard_channels>
<published_messages>910844</published_messages>
<stored_messages>63889</stored_messages>
<messages_in_trash>689483</messages_in_trash>
<channels_in_delete>0</channels_in_delete>
<channels_in_trash>0</channels_in_trash>
<subscribers>2269</subscribers>
<uptime>369649</uptime>

1. 16e6b4f7645dc6645ac81998af7649cd59328360
2. 1.13.3

3. 
   push_stream_shared_memory_size 2048M;
   push_stream_max_messages_stored_per_channel 30;
   push_stream_message_ttl 6h;
   push_stream_ping_message_text "{\"type\":\"heartbeat\"}";

server { listen 443 ssl http2; server_name tjournal.ru;

location ~ ^/secretpub/(.+) {
    internal;

    push_stream_publisher admin;
    push_stream_channels_path $1;
    push_stream_store_messages on;
}

location ~ ^/sub/(.+) {
    internal;

    push_stream_subscriber;
    push_stream_ping_message_interval 10s;
    push_stream_channels_path $1;
    push_stream_message_template ~text~\n;
    push_stream_store_messages on;
}

location ~ ^/ws/(.+) {
    internal;

    push_stream_subscriber websocket;
    push_stream_websocket_allow_publish on;
    push_stream_ping_message_interval 10s;
    push_stream_channels_path $1;
    push_stream_message_template ~text~\n;
    push_stream_subscriber_connection_ttl 20m;
    push_stream_store_messages on;
}

location ~ /chan/([\w\d\-_:/\.]+)$ {
    set $chan $1;

    if ($http_upgrade ~ "websocket") {
        rewrite ^ /ws/$chan last;
    }

    rewrite ^ $chan_prefix$chan last;
}

}

4. 

nginx version: nginx/1.13.3 built by gcc 4.9.2 (Debian 4.9.2-10) built with OpenSSL 1.0.2l 25 May 2017 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,-rpath,/usr/local/lib/' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-file-aio --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --add-module=/usr/src/lua-nginx-module-0.10.9rc8/ --add-module=/usr/src/nginx-push-stream-module-master/ --add-module=/usr/src/ngx_devel_kit-master/

[wandenberg]

@chekalskiy Try to disable the http2 support and check if the behavior stays the same. The module was developed before Nginx adds support to it and maybe they are incompatible. I don't have any other issue like that without http2.

[wandenberg]

Hi @chekalskiy, were you able to do this test?

[develar]

@wandenberg I can confirm that push_stream_store_messages directive doesn't work correctly for HTTP2. In my case if I push message to channel before client connect and consume, message is lost.

[wandenberg]

Hi @develar, as I said to @chekalskiy the module was developed before the http2. And as far as I can see, does not make sense to use both together, since the module already does streaming delivery and for posting, you can use keepalive connections. Do you see any use case to have both working together?

[develar]

@wandenberg Problem is that you cannot use HTTP2 on a server side. Nginx doesn't support HTTP2 proxy pass and not going to do because it doesn't make sense (you are right).

Yes — before using nginx my app was accessed by client directly using HTTP2 and SSE was not required. But now nginx is used as a frontend (better security, better functionality (limit requests, geolocation), better performance (nginx handle large uploads and only after full upload asks backend to handle file and after processing nginx serves large downloads back to client)).

So, nginx-push-stream-module is used to notify client about progress and results. Since nginx uses HTTP 1 to connect to backend, backend have to use HTTP 1 and cannot use http2 pushStream (as it was in a first iteration of my app). And nginx-push-stream-module play it's role very well.

For now, as workaround, I create channel before first upload request will be ended, so, client is able to subscribe to events as soon as upload request is ended — to not miss any event.

Workaround works, although there are 2 minor issues:

• in my case better to subscribe after 5 seconds or so because in most cases result will be not faster
• theoretically, there is a chance that backend will fire "error" event immediately after job added to queue and so.... event will be ignored. Very little chance, but anyway, not robust (that's why I use push_stream_store_messages).