search

ileitch / rapns

MOVED TO https://github.com/rpush/rpush
MIT License
291 stars 92 forks source link

ActiveRecord::ConnectionTimeoutError #196

Open j-mcnally opened 10 years ago

j-mcnally commented 10 years ago 
[2014-01-23 00:43:04] [ERROR] ActiveRecord::ConnectionTimeoutError, could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `acquire_connection'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:356:in `block in checkout'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:355:in `checkout'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record/reconnectable.rb:17:in `with_database_reconnect_and_retry'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record.rb:12:in `deliverable_notifications'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:42:in `enqueue_notifications'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:27:in `loop'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:27:in `feed_forever'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:11:in `block in start'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:28:in `block in feed_forever'

I am running on heroku and have been getting quite a bit of these, i think its a heroku issue but wanted confirm i dont need to be catching and reconnecting anywhere?

j-mcnally commented 10 years ago

The solution here was to add

ActiveRecord::ConnectionTimeoutError

to the ADAPTER_ERRORS in reconnectable.

Does this make anysense, without it the db wouldnt try the connection again

ileitch commented 10 years ago

Yes, checking for ConnectionTimeoutError does make sense. Usually this error is a symptom of not having a large enough connection pool. What is the sum of all connections in your rapns_apps table?

j-mcnally commented 10 years ago

i was afraid this was an issue, i have to set small DB_POOL size because we are wroking on heroku postgres_hobby which limits pool size to 20. Probably why I ran into this, while probably a bit slower, checking ConnectionTimeoutError seems to have resolved the symptoms.