403 forbidden 'Subscriber could not create channels.", but channels-stats shows channel active.

[pattist]

Pushstream is a great nginx module, really helpful.

However, I am having a problem with the 0.4 version: My application starts up correctly It receives all initial messages posted to the pushstream by a backend & displays them But... after a while, the pushstream longpolling client will begin receiving '403 forbidden' from the nginx pushstream server with an extended reason code of "X-Nginx-PushStream-Explain: Subscriber could not create channels." However, using curl 'http://localhost:6821/channels-stats?id=ALL*', nginx still thinks the channel is up and active, but it is issuing 403 to the longpolling subscriber.

Any ideas? Thanks for any help.

nginx config and subscriber javascript below.

NGINX Config

user root; worker_processes 1; worker_rlimit_core 500m; working_directory /var/log/nginx; error_log /var/log/nginx/error.log debug;

events { worker_connections 1024; }

http {

include       mime.types;
default_type  application/octet-stream;

access_log  /var/log/nginx/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

# denial of service limits
limit_zone   one  $binary_remote_addr  10m;

upstream adminapp {
    server 127.0.0.1:7000;
}

push_stream_shared_memory_size               60m;

push_stream_max_channel_id_length            200;

# max messages to store in memory
push_stream_max_messages_stored_per_channel  200;

# message ttl
push_stream_message_ttl                       5m;

# ping frequency
push_stream_ping_message_interval            10s;

# connection ttl to enable recycle
push_stream_subscriber_connection_ttl        4800s;

# connection ttl for long polling
push_stream_longpolling_connection_ttl      4800s;

push_stream_authorized_channels_only        on;

# push_stream_broadcast_channel_max_qtd        3;

# kill channel after 1 hour of inactivity.
push_stream_channel_inactivity_time 3600s;

server {

    listen 6821;
    server_name 127.0.0.1;

    location /pub {

        # Do not log internal Realview messages to nginx
        access_log off;

        # activate publisher mode for this location, with admin support
        push_stream_publisher admin;
        push_stream_channels_path               $arg_id;
        # query string based channel id
        #set $push_stream_channel_id             $arg_id;
        # store messages in memory
        push_stream_store_messages              on;
        # Message size limit
        client_max_body_size                    128k;
        client_body_buffer_size                 128k;
    }

    location /channels-stats {
        # activate channels statistics mode for this location
        push_stream_channels_statistics;
        push_stream_channels_path               $arg_id;
    }

}

server {
    listen      80 default_server;
    server_name xxx.example.com;

    client_max_body_size 5000m;

    error_page  404 405  /40x.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
 }

 location /stat {
       rtmp_stat all;
       rtmp_stat_stylesheet stat.xsl;
       #allow 127.0.0.1;
 }

 location /rawstat {
       rtmp_stat all;
 }

 location /stat.xsl {
        # you can move stat.xsl to a different location
        # under linux you could use /var/user/www for example
        root /home/wwwrun/public;
    }

 # Serve the assets files directly
 location ~ ^/assets/ {
    root /home/wwwrun/public/;
    expires 1y;
    add_header Cache-Control public;
    add_header ETag "";
    break;
 }

 location ~ ^/lobby/ {
       root   /home/wwwrun/public/;
 }
 location ~ ^/files/ {
       root   /home/wwwrun/public/;
 }

 location /backups {
    root /home/wwwrun/data;
    internal;
 }

 # public long-polling endpoint
 location ~ /lp/(.*) {

    # activate long-polling mode for this location
    push_stream_subscriber      long-polling;
    push_stream_channels_path   $1;
    push_stream_last_received_message_tag       $arg_tag;
    push_stream_last_received_message_time      $arg_time;

    # message template
    push_stream_message_template                "{\"id\":~id~,\"channel\":\"~channel~\",\"tag\":\"~tag~\",\"time\":\"~time~\",\"text\":~text~}";

    # connection timeout
    push_stream_longpolling_connection_ttl        4800s;
  }

  location / {
     proxy_pass  http://adminapp;
     proxy_redirect     off;
     proxy_set_header   Host $host:$server_port;
     proxy_set_header   X-Real-IP        $remote_addr;
     proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
     proxy_next_upstream off;
     proxy_read_timeout 300;
     limit_conn   one  25;  # denial of service limits

     proxy_set_header  X-Sendfile-Type   X-Accel-Redirect;
     proxy_set_header  X-Accel-Mapping       /home/wwwrun/data/=/nginx_direct_data/;

  }

  location /nginx_direct_data/ {
     internal;
     alias   /home/wwwrun/data/;
  }

}

}

javascript for pushstream

  var pushstream = null;
  var perform_unload = true;
  var ready_once = false;

  function swfLoadComplete(e) {
    if (!e.success) {
      alert("Real View Load failed.")
    }
  }

  function myCometCallback(response) {
    try {
      if (response != '') {
        //var json = jQuery.parseJSON( response );
        if (response.type == 'channel_kill') {
          perform_unload = false;
          pushstream.disconnect();
          setTimeout(function(){
            window.location.reload();
          },20000);
          return;
        }
        if (response.type == 'create') {
          // just ignore for now
          return;
        }
        if (response.type == 'keep_alive') {
          // just ignore
          return;
        }
        var obj = swfobject.getObjectById("rvApp");
        var msg = JSON.stringify(response);
        obj.cometMessage( msg );
      }
    } catch(e) { /* ignore */ }
  }

  // Called by the flex actionscript when it is
  // loaded and completes running its init0 function. 
  function flexIsReady() {
    //var chan         = "real_300";
    var stop_url     = "/realview/stopobserving/300";
    var obj = swfobject.getObjectById("rvApp");

    obj.init1( "300", AUTH_TOKEN ); 
    // watch for page unload. when it happens stop channel and unload swf
    $(window).unload(function() {
      try {
        // when this page unloads
        // make a good faith effort to stop the channel, but...
        // ignore any errors.
        if (perform_unload) {
          $.get( stop_url );
        }
      }
      catch(error) { /* ignore */ }
      swfobject.removeSWF("rvApp");
    });

  };

  function flexIsReadyForMessages() {

    pushstream = new PushStream({
      useSSL: false,
      host: window.location.hostname,
      port: window.location.port,
      modes: "longpolling",
      messagesControlByArgument: true,
      tagArgument: 'tag',    
      timeArgument: 'time' 
    });
    // start the message consume loop
    pushstream.onmessage = myCometCallback;
    pushstream.onstatuschange = statuschanged;
    pushstream.addChannel('real_300');
    pushstream.connect();

  };

  function statuschanged(state) {
    if (state == PushStream.OPEN) {
      // tell edrv to start sending
      if (!ready_once) {
        $.get( "/realview/observer_ready/300" );
        ready_once = true;
      }
    }
  }

[wandenberg]

Hi @pattist

as you set push_stream_authorized_channels_only to on, the subscriber is only allowed to connect in a preexisting channel with at least one message. And as you set push_stream_message_ttl to 5m, when has elapsed 5m since the last published message the server will not accept subscribers to this channel. But the channel will completely disappear from your server only after 1 hour, since you set the push_stream_channel_inactivity_time to 3600s.

The behavior is as expected/designed for.

This was done to be possible to "terminate a subject". If no one is publishing a message to a channel, so this topic can be considered as finished. And to be possible collect the memory used by this channel we can't allow new connections.

I hope this info helps you.

[pattist]

Hi @wandenberg

Thanks for the quick response.

I didn't understand the use and interaction of the 'ttl' and 'inactivity' timers. I think the solution for me is to send 'keepalive' messages from the publisher side, which are ignored by the client.

My application is long lived, but the messages can be infrequent or they can be high volume depending on activity in an external server application. Messages older than 5 minutes are not useful to the client and if nothing happens within an hour then there probably has been a failure in an external app or the publisher app. I set the 1 hour mainly as a way for nginx-pushstream to recover resources on an unexpected failure.

[wandenberg]

You can set the message ttl with a greater value, like 30 minutes and configure your client with secondsAgo: 300 with that when a new subscriber connect for the first time it will get only messages published with less than 5 minutes, but will allow users connect to the channel for more time. Only blocking the access if there was more than 30 minutes without a published message

[pattist]

Thanks for the help