ConnectionPool in a gem + rails 3 + sidekiq

[florrain]

Hello,

First thanks for this library which is apparently very useful especially with Sidekiq. I'm saying apparently because I haven't tested it yet in a production scenario and would like to have your feedback before that.

I'm using it inside an external gem called responsys_api which is an api client for the Oracle Responsys API. The ConnectionPool is wrapped in a SessionPool here. It is used by the client here. The branch is not merged yet but most of the logic is pushed. I'd make some changes to the code if you have any concern.

Then in the infrastructure we have at thredUP, we make async calls using the client gem inside sidekiq jobs. Because sometimes we create hundreds of thousands jobs but with only 100 connections, we want to make sure we're not exceeding and blocking our account and this is where your gem helps.

So two questions:

• The connection_pool is not instanciated from inside a worker unlike the sidekiq wiki example but when the client gem is being configured in a rails initializer. The pool is then used by the client gem when the job makes a call. Question is: do you have any idea if the gem will work in the same way and will not loose its benefit of managing one alone pool for all the jobs ? I'm thinking of different threads issues and so different instances of the ConnectionPool because of the declaration scope. For me the variable will be global since it is in an initializer but...
• Any chance to set an unlimited timeout ? (timeout = 0). I'm thinking of a scenario with sidekiq where the number of processes would be way greater than the maximum number of api connections we're allowed to make at anytime and so the size of the pool. Because of the delta between these two can put a certain number of jobs waiting for the pool and the difficulty to evaluate the timeout in seconds I thought of disabling the timeout. What do you think about this ? Any other workaround that would be "cleaner" ?

If it can help an example of the code:

#config/initializers/responsys.rb in the rails app
Responsys.configure do |config|
  config.settings = {
    username: "user",
    password: "password",
    wsdl: "http://oracle.soap.wsdl",
    sessions: { timeout: 1200 } #params passed to the connection_pool in the internal SessionPool object
  }
end

#async job enqueued when the user updates his emailing options
class UserResponsysStatusJob
  include Sidekiq::Worker
  sidekiq_options :queue => :responsys

  def perform(email)
    Responsys::Member.new(email).subscribe("mailing_list") #Uses the gem to make the call after picking up a client in the pool.
  end
end

Also do you think of any integration in the sidekiq UI ? Would be cool to real-time monitor if there's a registered pool to know how many clients are active in the pool.

Thanks for your help!