search

documentcloud / cloud-crowd

Parallel Processing for the Rest of Us
https://github.com/documentcloud/cloud-crowd/wiki
MIT License
851 stars 92 forks source link

AR connection pool exhaustion #43

Closed knowtheory closed 9 years ago

knowtheory commented 10 years ago

Because the CloudCrowd server forks off processes using Thread.new in order to prevent the server from blocking, new threads claim an AR connection, but never notify the connection pool when the thread is finished with a connection.

Consequently, if more jobs are quickly submitted than there are connections in the pool, the server freaks out at the empty connection pool. A similar problem is documented in a variety of places:

The solution we're going with is providing an abstraction that takes a block, wraps it inside an explicit AR connection inside of a thread as mperham points out here: https://github.com/mperham/sidekiq/issues/1047#issuecomment-20643026

It'll probably be called CloudCrowd.defer.

nathanstitt commented 9 years ago

This was fixed in 38aaaef69ccada56f511e18f78eb9ffc6fac4b77, and has been running without issues for several months now, closing the issue.