Pubsubstub improvment

[byroot]

Currently Pubsubstub cause a bunch of minor but annoying issues.

• Any other server than thin will run it in "degraded" mode
• Even with thin, you have to force reload (closing open connections), otherwise SSE connections will never timeout

I discussed this a bit with @gmalette, and we think it would be valuable to be able to run it in an external process. That way force reloading it is not an issue.

The default would still be to run it inline though.

[davidcornu]

@byroot can we gracefully close the open connections?

[byroot]

Oh right I forgot to mention that. So I'd need to search for the github issues and all again. But AFAIK if you use Sinatra streaming helper, there is no known way to gracefully close the connection.

[davidcornu]

Ah so the "solution" would be to use rack.hijack and manually maintain an open connection pool using EM stuff. Sounds painful.

[gmalette]

Is this finished?

[casperisfine]

Well, we do run it externally at Shopify. But it requires quite a bit of custom code to do so.

We should make it trivial before we close this issue.

[jnraine]

We're experiencing pain from this as well.

@casperisfine, are you able to share the custom code Shopify uses to make this run externally? Even though it is non-trivial, it may help us resolve the problem on our end. Thanks!

[casperisfine]

Sure:

Procfile:

stream: "bin/puma --config pubsubstub/puma_config.rb pubsubstub/config.ru --port 8000 --environment $RAILS_ENV"

pubsubstub/puma_config.rb:

#!/usr/bin/env puma

# Tells puma to not wait on clients to sutdown
force_shutdown_after 0

# Number of processes
workers 2

# min, max threads per workers
threads 32, 256

pubsubstub/config.ru:

require 'yaml'
require 'uri'
require 'rack'
require 'rack/session/redis'
require 'pubsubstub'
require 'bugsnag'

require_relative '../lib/user_required_middleware'

module AppConfig
  PATH = File.expand_path('../../config/secrets.json', __FILE__)
  extend self

  def redis_url
    url = URI.parse(config.fetch('redis_url', 'redis://localhost'))
    url.port ||= 6379 # EM::Redis is stupid, it need an explicit port
    url.path = '/5/session'
    url.to_s
  end

  def redis_session
    {redis_server: redis_url, key: "_shipit_session_id_#{RUBY_VERSION}"}
  end

  def config
    @config ||= JSON.load(File.read(PATH).gsub('${HOST_IP}', ENV.fetch('HOST_IP', 'localhost')))
  end

  def environment
    ENV['RAILS_ENV'] || ENV['RACK_ENV'] || 'development'
  end
end

class HealthCheck
  def initialize(app)
    @app = app
  end

  def call(env)
    return @app.call(env) unless env['PATH_INFO'] == '/status/version'
    [200, {'Content-Type' => 'text/plain'}, %w(OK)]
  end
end

puts "Connecting to #{AppConfig.redis_url}"
Pubsubstub.redis_url = AppConfig.redis_url
Pubsubstub.error_handler = -> (error) { Bugsnag.notify(error) }

use Rack::Session::Redis, AppConfig.redis_session
use HealthCheck
use UserRequiredMiddleware
run Pubsubstub::StreamAction.new

Most of the pubsubstub/config.ru is a bit shopify specific though. You basically only need:

require 'rack'
require 'pubsubstub'

Pubsubstub.redis_url = 'something'
run Pubsubstub::StreamAction.new

Except the endpoint can end up public depending what your auth strategy is.

Additionally we have a nginx route to forward the SSE traffic to that separate process pool:

  location /events {
    proxy_pass http://shipit-stream;
    rewrite /events(.*) /$1  break;

    proxy_set_header Host               $http_host;
    proxy_set_header Client-IP          $remote_addr;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_set_header X-Forwarded-Proto  https;
    proxy_set_header X-Forwarded-Port   443;
    proxy_set_header X-Request-Start    "t=${msec}000";

    proxy_redirect off;
    proxy_read_timeout 40s;
    proxy_buffering off;
  }

Hope this helps.

[jnraine]

@casperisfine thank you! 💯

[airhorns]

Is this still the recommended way to getting pub sub to work nice?

[casperisfine]

Yup!

[ohsabry]

@casperisfine It would be really helpful (and safe) if you can provide some rack-idiot-proof guidance on how to implement use UserRequiredMiddleware (which is from pre-engine days) to check the session that was created from github auth here

[casperisfine]

It depends on what authentication method you have enabled. But the one you link should work fine for most people.