search

influitive / apartment

Database multi-tenancy for Rack (and Rails) applications
2.66k stars 460 forks source link

Rails 5: ActiveRecord::ConnectionNotEstablished: No connection pool with id primary found #650

Open AkshayGoyal022 opened 4 years ago

AkshayGoyal022 commented 4 years ago

We have 2 servers: master and a slave server. slave server is primarily being used to handle reports. All the long background running processes are pushed to the slave server. We are using sidekiq for background processing.

Now, whenever a report is ready we create a Download object in which the generated report is attached. As the process is running in slave we need to connect to master in order to create Download object. Code looks something like this:

def perform(tenant_name)
  Apartment::Tenant.switch(tenant_name) do
    # data processing code here....
    connection = MasterDbConnectionService.connect(tenant_name)
    Download.create!(time_taken: time_taken, file: file) # Pseudo code
    # Disconnect from the previously established connection
    MasterDbConnectionService.disconnect(connection)
  end
end

But we are intermittently facing the error especially for long running jobs most probably because of sidekiq retrying that job.

ActiveRecord::ConnectionNotEstablished) "No connection pool with 'primary' found.

MasterDbConnectionService code:

def connect(tenant_name)
  ActiveRecord::Base.establish_connection(master_config.merge(schema_search_path: "'#{tenant_name}',public,shared_extensions"))
end

def  disconnect(connection)
  ActiveRecord::Base.remove_connection(connection)
end

Config: Nginx, Phusion Passenger, Postgres, ruby-2.3.1, rails 5.1.3

Full stack trace:

/home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activerecord-5.1.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:930:in `retrieve_connection'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activerecord-5.1.3/lib/active_record/connection_handling.rb:116:in `retrieve_connection'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activerecord-5.1.3/lib/active_record/connection_handling.rb:88:in `connection'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/apartment-2.1.0/lib/apartment/adapters/postgresql_adapter.rb:40:in `reset'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/apartment-2.1.0/lib/apartment/adapters/abstract_adapter.rb:88:in `rescue in ensure in switch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/apartment-2.1.0/lib/apartment/adapters/abstract_adapter.rb:88:in `ensure in switch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/apartment-2.1.0/lib/apartment/adapters/abstract_adapter.rb:88:in `switch'
  /home/deployer/workspace/rails/dms_webapp/app/jobs/report_download_job.rb:12:in `perform'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/execution.rb:37:in `block in perform_now'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/i18n-0.9.5/lib/i18n.rb:268:in `with_locale'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `instance_exec'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/logging.rb:24:in `block (4 levels) in <module:Logging>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/notifications.rb:166:in `block in instrument'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/notifications.rb:166:in `instrument'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/logging.rb:23:in `block (3 levels) in <module:Logging>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/logging.rb:44:in `block in tag_logger'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:69:in `block in tagged'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:26:in `tagged'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:69:in `tagged'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/logging.rb:44:in `tag_logger'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/logging.rb:20:in `block (2 levels) in <module:Logging>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `instance_exec'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:135:in `run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/execution.rb:33:in `perform_now'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/execution.rb:22:in `block in execute'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/railtie.rb:26:in `block (4 levels) in <class:Railtie>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/execution_wrapper.rb:85:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/reloader.rb:68:in `block in wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/execution_wrapper.rb:81:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/reloader.rb:67:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/railtie.rb:25:in `block (3 levels) in <class:Railtie>'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `instance_exec'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/callbacks.rb:135:in `run_callbacks'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/execution.rb:20:in `execute'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activejob-5.1.3/lib/active_job/queue_adapters/sidekiq_adapter.rb:40:in `perform'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:199:in `execute_job'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:133:in `invoke'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:169:in `block in process'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:97:in `local'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/rails.rb:42:in `block in call'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/execution_wrapper.rb:85:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/reloader.rb:68:in `block in wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/execution_wrapper.rb:85:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/activesupport-5.1.3/lib/active_support/reloader.rb:67:in `wrap'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/rails.rb:41:in `call'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:215:in `stats'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/job_logger.rb:7:in `call'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:72:in `global'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:129:in `block in dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:44:in `with_context'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:128:in `dispatch'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:168:in `process'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:85:in `process_one'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
  /home/deployer/.rvm/gems/ruby-2.3.1@rails513/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
MarceloCajueiro commented 4 years ago

Same thing happening here on sidekiq.

Sometimes it locks the threads too.

I was trying to replace https://github.com/sobrinho/activerecord-connections to Apartment because the other gem is too old and does not deal well with too many databases (it does not closes the connections properly)