PG Error: PG::UnableToSend

[workmaster2n]

workmaster2n/parallel@1d5eae391137d1fb58a5334a844734e480b8ca15

When I run

bundle exec ruby spec/cases/map_with_ar_postges.rb 

I get back:

making user
making user
making user
making user
making user
making user
making user
making user
/Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:512:in `exec': server closed the connection unexpectedly (PG::UnableToSend)
    This probably means the server terminated abnormally
    before or while processing the request.
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:512:in `dealloc'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:495:in `block in clear'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:494:in `each_value'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:494:in `clear'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:557:in `clear_cache!'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_adapter.rb:322:in `reconnect!'
    from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:569:in `reconnect!'
    from spec/cases/map_with_ar_postges.rb:39:in `block in <main>'
    from /Users/tyler/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/tempfile.rb:320:in `open'
    from spec/cases/map_with_ar_postges.rb:4:in `<main>'

It appears that something is going awry with postgres.

[grosser]

Uhh nice, was never able to reproduce this crazyness on mysql :)

[grosser]

Maybe try the reconnect inside the block or in various other places

Parallel.each(...) do
  @reconnected ||= User.connection.reconnect! || true
  ...
end

[workmaster2n]

workmaster2n/parallel@d9686f45f85506a8d88d0f34d6bd793e06f5059f

Tried all 6 combinations (each/map, in, after, in/after). Couldn't get any different result.

What should the output of User.connection.reconnect!.inspect be?

[grosser]

Last time I tried it was nil, so I added the || true part.

Not sure how to fix that, maybe there is some more general answer out there for postgres vs fork vs closed connection :/

Last time we had this problem we loaded everything before forking :D

On Thu, Oct 10, 2013 at 6:22 AM, Tyler DeWitt notifications@github.comwrote:

workmaster2n/parallel@d9686f4https://github.com/workmaster2n/parallel/commit/d9686f45f85506a8d88d0f34d6bd793e06f5059f

Tried all 6 combinations (each/map, in, after, in/after). Couldn't get any different result.

What should the output of User.connection.reconnect!.inspect be?

— Reply to this email directly or view it on GitHubhttps://github.com/grosser/parallel/issues/83#issuecomment-26052888 .

[jschroeder9000]

This issue was a bear. My use case for this gem is importing data from a legacy Oracle database to a new postgres database. I want the import to run as fast as possible and forking concurrent processes was an obvious way to speed things up. There are a variety of import functions that all call a private process function that (initially) looked something like:

def process (collection, options={})
  return if collection.size == 0

  options[:title] ||= collection.first.class.to_s.titleize.split('/').last.pluralize
  options[:parallel] = true unless options.has_key?(:parallel)
  progress_bar = ProgressBar.create(title: options[:title], total: collection.size, format: '%t: |%B| %c/%C (%P%%)%E')
  if options[:parallel]
    Parallel.each(collection, finish: lambda{|item, index, result| progress_bar.increment}) do |item|
      yield item
    end
  else
    collection.each do |item|
      yield item
      progress_bar.increment
    end
  end
  progress_bar.finish
end

This worked great until one of the import functions I wrote caused the issue mentioned here. Everything up to that point worked fine, but the next interaction with postgres would reliably fail. I tried the suggestions here and in the readme everywhere I could think of, but got nowhere. Running everything without parallel worked fine so I thought 'Okay, parallel must be doing something funky... I'll try something simple', which brought me to something like this:

def process (collection, options={})
  return if collection.size == 0

  options[:title] ||= collection.first.class.to_s.titleize.split('/').last.pluralize
  options[:parallel] = true unless options.has_key?(:parallel)
  progress_bar = ProgressBar.create(title: options[:title], total: collection.size, format: '%t: |%B| %c/%C (%P%%)%E')
  if options[:parallel]
    pids = []
    num_processes = Parallel.processor_count
    collection.in_groups(num_processes, false) do |group|
      pids << Process.fork do
        group.each do |item|
          yield item
          progress_bar.increment
        end
      end
    end
    pids.each do |pid|
      Process.waitpid(pid)
    end
  else
    collection.each do |item|
      yield item
      progress_bar.increment
    end
  end
  progress_bar.finish
end

But that also failed reliably at the same point that using parallel did. I really didn't want to give up on parallel processing for this, so I eventually came upon #62 (which I think is the same issue as this) and the comment from @LFDM. Finally I had an implementation that would get past the point it was failing with:

def process (collection, options={})
  return if collection.size == 0

  options[:title] ||= collection.first.class.to_s.titleize.split('/').last.pluralize
  options[:parallel] = true unless options.has_key?(:parallel)
  progress_bar = ProgressBar.create(title: options[:title], total: collection.size, format: '%t: |%B| %c/%C (%P%%)%E')
  if options[:parallel]
    Parallel.each(collection, finish: lambda{|item, index, result| progress_bar.increment}) do |item|
      yield item
    end

    begin
      ActiveRecord::Base.connection.reconnect!
    rescue
      ActiveRecord::Base.connection.reconnect!
    end
  else
    collection.each do |item|
      yield item
      progress_bar.increment
    end
  end
  progress_bar.finish
end

Kind of a long post, but the fact that I could reproduce the same problem without parallel (but not without forking processes) indicates that there is some kind of issue when forking processes with activerecord and postgres and that this is not a problem caused by parallel.

A hackaround is better than nothing and I'm glad I found @LFDM's post in #62 - I never would have thought to retry reconnecting a second time (have I ever told you the definition of insanity?), but it would still be nice to know what the heck is going on.

Hope that helps someone. Maybe it's worth considering adding @LFDM's suggestion to the readme since it actually works?

[gnagel]

@jschroeder9000 Thanks for the detailed breakdown. This helped me a lot!

[grosser]

any way we can make this work better out of the box ?

[grosser]

fyi you can now use :progress => options[:title] instead of building progressbar by hand

[duffyjp]

I'm running into this issue following a MySQL -> Postgres conversion. I use Parallel extensively in rendering views by silently generating fragment caches with Parallel then a normal pass to render them out of memcached. I use this pattern enough I wrote a gem for it. Here's the relevant helper: https://github.com/duffyjp/duffy/blob/master/lib/duffy/beast_mode_helper.rb

This all works fine in MySQL, but in Postgres with the same application and same data I get the errors listed here and the behavior in #62.

PG::UnableToSend: no connection to the server

ActiveRecord::StatementInvalid (PG::ConnectionBad: PQsocket() can't get socket descriptor:

I've tried the three README workarounds, and even the quite hideous

    begin
      ActiveRecord::Base.connection.reconnect!
    rescue
      ActiveRecord::Base.connection.reconnect!
    end

... and nothing has worked.

Has there been any progress on this issue in the last couple years?

	Version
pg	0.18.4
parallel	1.6.2
PostgreSQL	9.5.1
Ruby	2.2.3
Rails	4.2.5.1

[grosser]

status afaik is that there are various workarounds for mysql ... not sure which one is best ... no idea about postgres ... ideally find 1 solution that works and codify that so we stop copy-pasting things and can start writing test / finding bugs

[duffyjp]

status afaik is that there are various workarounds for mysql

The funny thing is with mysql I've never once needed to use any of those workarounds. It's always worked for me out of the box. Same with Oracle. I even do both at the same time, reading from Oracle, writing to mysql, 16 threads at once.

Do you have any gut feeling where the problem might be?

[duffyjp]

I started writing an issue on the pg gem's bitbucket page, but following their guide I think it might be more productive to start here again.

Here's what I found:

postgres + thread = okay postgres + AR + thread = okay postgres + AR + Parallel => connection lost...

I built some minimal test scripts following the pg's issue tracker guide which I'll post here:

pg_thread_test.rb

#!/usr/bin/env ruby

require 'pg'

conn = PG.connect( :dbname => 'test', user: 'test', password: 'test' )
$stderr.puts '---',
    RUBY_DESCRIPTION,
    PG.version_string( true ),
    "Server version: #{conn.server_version}",
    "Client version: #{PG.respond_to?( :library_version ) ? PG.library_version : 'unknown'}",
    '---'

result1 = conn.exec( "SELECT * FROM people LIMIT 1;" )
$stderr.puts "Non Threaded:"
$stderr.puts %Q{Expected this to return: ["1"]}
p result1.field_values('id')

t = Thread.new do
      result2 = conn.exec( "SELECT * FROM people LIMIT 1;" )
      puts "In the Thread:"
      puts %Q{Expected this to return: ["1"]}
      p result2.field_values('id')   
end
t.join()

result3 = conn.exec( "SELECT * FROM people LIMIT 1;" )
$stderr.puts "Repeat Non Threaded:"
$stderr.puts %Q{Expected this to return: ["1"]}
p result3.field_values('id')

Output:

$ruby pg_thread_test.rb 
---
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
PG 0.18.4 (build da42b972b5ab)
Server version: 90501
Client version: 90501
---
Non Threaded:
Expected this to return: ["1"]
["1"]
In the Thread:
Expected this to return: ["1"]
["1"]
Repeat Non Threaded:
Expected this to return: ["1"]
["1"]

[duffyjp]

pg_ar_thread_test.rb

#!/usr/bin/env ruby

require 'pg'
require 'active_record'

$stderr.puts '---',
    RUBY_DESCRIPTION,
    PG.version_string( true ),
    "ActiveRecord version: #{ActiveRecord.version}",
    "Client version: #{PG.respond_to?( :library_version ) ? PG.library_version : 'unknown'}",
    '---'

class Person < ActiveRecord::Base
  establish_connection(
    adapter:  "postgresql",
    host:     "localhost",
    username: "test",
    password: "test",
    database: "test"
  )
end

puts "Non Threaded:"
puts "Expect: 1"
puts Person.first.id

t = Thread.new do
      puts "In the Thread:"
      puts "Expect: 1"
      puts Person.first.id
end
t.join()

puts "Repeat Non Threaded:"
puts "Expect: 1"
puts Person.first.id

Output:

$ ruby pg_ar_thread_test.rb 
---
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
PG 0.18.4 (build da42b972b5ab)
ActiveRecord version: 4.2.5.1
Client version: 90501
---
Non Threaded:
Expect: 1
1
In the Thread:
Expect: 1
1
Repeat Non Threaded:
Expect: 1
1

[duffyjp]

pg_ar_parallel_test.rb

#!/usr/bin/env ruby

require 'pg'
require 'active_record'
require 'parallel'

$stderr.puts '---',
    RUBY_DESCRIPTION,
    PG.version_string( true ),
    "ActiveRecord version: #{ActiveRecord.version}",
    "Parallel version: #{Parallel::VERSION}",
    "Client version: #{PG.respond_to?( :library_version ) ? PG.library_version : 'unknown'}",
    '---'

class Person < ActiveRecord::Base
  establish_connection(
    adapter:  "postgresql",
    host:     "localhost",
    username: "test",
    password: "test",
    database: "test"
  )
end

puts "Non Threaded:"
puts "Expect: 1"
puts Person.first.id

puts "Parallel Gem:"
puts "Expect 1,2,3 in any order"

Parallel.each(Person.limit(3), in_processes: 3) do |person|
  puts person.id
end

puts "Repeat Non Threaded:"
puts "Expect: 1"
puts Person.first.id

Output:

$ ruby pg_ar_parallel_test.rb
---
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
PG 0.18.4 (build da42b972b5ab)
ActiveRecord version: 4.2.5.1
Parallel version: 1.6.2
Client version: 90501
---`
Non Threaded:
Expect: 1
1
Parallel Gem:
Expect 1,2,3 in any order
1
2
3
Repeat Non Threaded:
Expect: 1
/Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec': PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (ActiveRecord::StatementInvalid)
    This probably means the server terminated abnormally
    before or while processing the request.
: SELECT  "people".* FROM "people"  ORDER BY "people"."id" ASC LIMIT 1
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:355:in `select'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:39:in `find_by_sql'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:639:in `exec_queries'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:515:in `load'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:243:in `to_a'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:500:in `find_nth_with_limit'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:484:in `find_nth'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:127:in `first'
from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:3:in `first'
    from pg_ar_parallel_test.rb:45:in `<main>'

[grosser]

Multithreading might work in newer mysql versions, did not need to use parallel db access in quiet a while ... but forking never seemed to work well.

We could start by having a few tests that show how it does work (with pg/sqlite/mysql) and then when someone runs into a bug they can add to it.

On Thu, Mar 24, 2016 at 6:59 AM, Jacob Duffy notifications@github.com wrote:

status afaik is that there are various workarounds for mysql

The funny thing is with mysql I've never once needed to use any of those workarounds. It's always worked for me out of the box. Same with Oracle. I even do both at the same time, reading from Oracle, writing to mysql, 16 threads at once.

Do you have any gut feeling where the problem might be?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/grosser/parallel/issues/83#issuecomment-200849722

[duffyjp]

Okay, I didn't expect this:

I tried the last file, but with mysql instead, and it fails... Even though similar statements work just peachy in my full-stack Rails app. Maybe there's another component at play.

mysql_ar_parallel_test.rb

#!/usr/bin/env ruby

require 'mysql2'
require 'active_record'
require 'parallel'

$stderr.puts '---',
    RUBY_DESCRIPTION,
    "MySQL version: #{}",
    "ActiveRecord version: #{ActiveRecord.version}",
    "Parallel version: #{Parallel::VERSION}",
    '---'

class Person < ActiveRecord::Base
  establish_connection(
    adapter:  "mysql2",
    host:     "localhost",
    username: "test",
    password: "test",
    database: "test"
  )
end

puts "Non Threaded:"
puts "Expect: 1"
puts Person.first.id

puts "Parallel Gem:"
puts "Expect 1,2,3 in any order"

Parallel.each(Person.limit(3), in_processes: 3) do |person|
  puts person.id
end

puts "Repeat Non Threaded:"
puts "Expect: 1"
puts Person.first.id

Output:

---
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
MySQL version: 
ActiveRecord version: 4.2.5.1
Parallel version: 1.6.2
---
Non Threaded:
Expect: 1
1
Parallel Gem:
Expect 1,2,3 in any order
12

3
Repeat Non Threaded:
Expect: 1
/Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:107:in `_query': Mysql2::Error: MySQL server has gone away: SELECT  `people`.* FROM `people`  ORDER BY `people`.`id` ASC LIMIT 1 (ActiveRecord::StatementInvalid)
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:107:in `block in query'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:106:in `handle_interrupt'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:106:in `query'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:305:in `block in execute'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:305:in `execute'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:355:in `select'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:39:in `find_by_sql'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:639:in `exec_queries'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:515:in `load'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:243:in `to_a'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:500:in `find_nth_with_limit'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:484:in `find_nth'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:127:in `first'
    from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:3:in `first'
    from pg_ar_parallel_mysql_test.rb:39:in `<main>'

Edit:

Workaround # 1 does work for this

Parallel.each(Person.limit(3), in_processes: 3) do |person|
  puts person.id
end
ActiveRecord::Base.connection.reconnect!

I'll try in Postgres this afternoon.

[grosser]

I think this is because the connection is taken into the fork and then closed when the fork ends ... So we'll have to reconnect after the forking is done ... Can you turn this into a testcase ?

On Thu, Mar 24, 2016 at 9:00 AM, Jacob Duffy notifications@github.com wrote:

Okay, I didn't expect this:

I tried the last file, but with mysql instead, and it fails... Even though similar statements work just peachy in my full-stack Rails app. Maybe there's another component at play. mysql_ar_parallel_test.rb

!/usr/bin/env ruby

require 'mysql2'require 'active_record'require 'parallel' $stderr.puts '---', RUBY_DESCRIPTION, "MySQL version: #{}", "ActiveRecord version: #{ActiveRecord.version}", "Parallel version: #{Parallel::VERSION}", '---' class Person < ActiveRecord::Base establish_connection( adapter: "mysql2", host: "localhost", username: "test", password: "test", database: "test" )end

puts "Non Threaded:" puts "Expect: 1" puts Person.first.id

puts "Parallel Gem:" puts "Expect 1,2,3 in any order" Parallel.each(Person.limit(3), in_processes: 3) do |person| puts person.idend

puts "Repeat Non Threaded:" puts "Expect: 1" puts Person.first.id

Output:

---

ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14] MySQL version: ActiveRecord version: 4.2.5.1

Parallel version: 1.6.2

Non Threaded: Expect: 1 1 Parallel Gem: Expect 1,2,3 in any order 12

3 Repeat Non Threaded: Expect: 1 /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:107:in _query': Mysql2::Error: MySQL server has gone away: SELECTpeople.* FROMpeopleORDER BYpeople.idASC LIMIT 1 (ActiveRecord::StatementInvalid) from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:107:inblock in query' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:106:in handle_interrupt' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/mysql2-0.4.3/lib/mysql2/client.rb:106:inquery' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:305:in block in execute' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:472:inblock in log' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in instrument' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:466:inlog' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:305:in execute' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:inexecute' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in exec_query' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:355:inselect' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in select_all' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:inselect_all' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:39:in find_by_sql' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:639:inexec_queries' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:515:in load' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation.rb:243:into_a' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:500:in find_nth_with_limit' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:484:infind_nth' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:127:in first' from /Users/duffyjp/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:3:infirst' from pg_ar_parallel_mysql_test.rb:39:in `

'

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/grosser/parallel/issues/83#issuecomment-200901221

[parreirat]

Had same issue. From the PSQL manual:

Warning On Unix, forking a process with open libpq connections can lead to unpredictable results because the parent and child processes share the same sockets and operating system resources. For this reason, such usage is not recommended, though doing an exec from the child process to load a new executable is safe.

Source: https://www.postgresql.org/docs/9.0/static/libpq-connect.html

[parreirat]

Given this, here's the way I've dealt with using PSQL databases with multiple processes: (works perfectly)

def reconnect_database
  config = ActiveRecord::Base.connection.instance_variable_get("@config").stringify_keys!
  ActiveRecord::Base.establish_connection(config)
end

ActiveRecord::Base.connection.disconnect!
(1..process_number).each do |process_index|
  processes << Process.fork do
    reconnect_database
    # Do stuff
  end
end
Process.waitall
reconnect_database

[jjb]

just wanted to say that I've been debugging something related for hours and this...

I think this is because the connection is taken into the fork and then closed when the fork ends

...was a huge aha moment for me. yay! 😹