grosser / parallel

Ruby: parallel processing made simple and fast
MIT License
4.16k stars 254 forks source link

Limit Max DB Connections in Threads #289

Closed berniechiu closed 3 years ago

berniechiu commented 3 years ago

Hi all,

I wonder if it's possible to limit DB connections to 1 or same connection with thread without raising connection pool error which we mainly use for parallel HTTP requests but might cause extra connection leaks if we wrap something more than just HTTP connections.

Ex.

class ExampleExecutor
  def self.process(url, shipment = nil)
    response = HTTParty.post(url)
    # Don't want this one to open new connection, it's ok to share with the top level DB connection
    shipment.touch if shipment
    response
  end
end

responses = Parallel.map([url1, url2, url3]) do |url|
  ExampleExecutor.process(url, shipment)
end
grosser commented 3 years ago

it's not ok to share the with the local connection, there are race conditions inside of AR / it's not threadsafe afaik

berniechiu commented 3 years ago

@grosser thanks for the suggestion 😄