Rails 6.0.3: *** stack smashing detected ***

[jfahrenkrug]

Hi,

we have an old-ish Rails application that we are trying to upgrade to Rails 6. We have everything working, but we have a couple of spots where we use Parallel.map to iterate over chunks of DB ids so we can execute sub-queries in parallel (it's a complex data model and this was done to speed up the initial generation of a large JSON response that is subsequently cached). This worked fine in Rails 4 with Unicorn, but in Rails 6 we get this error (tried it both with Puma and with Unicorn):

#<Thread:0x000055ffc2124090 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffc21243b0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffbd8db9a0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
#<Thread:0x000055ffc2124220 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep'/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': : stack level too deepstack level too deep ( (SystemStackErrorSystemStackError))

    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffc2124540 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
#<Thread:0x000055ffc21246d0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffbd8dab90 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
    from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
I, [2022-02-10T18:49:43.804366 #14]  INFO -- : [461a9905-5681-47f9-b18a-59048d3e1310] Completed 500 Internal Server Error in 63460ms (ActiveRecord: 2768.6ms | Allocations: 106815)
F, [2022-02-10T18:49:43.805046 #14] FATAL -- : [461a9905-5681-47f9-b18a-59048d3e1310]
[461a9905-5681-47f9-b18a-59048d3e1310] Parallel::DeadWorker (Parallel::DeadWorker):
[461a9905-5681-47f9-b18a-59048d3e1310]
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:78:in `rescue in work'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:75:in `work'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:405:in `block (4 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:518:in `with_instrumentation'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:404:in `block (3 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:392:in `loop'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:392:in `block (2 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:215:in `block (4 levels) in in_threads'
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
I, [2022-02-10T18:49:46.981924 #15]  INFO -- : [ebab2068-1248-466e-980c-f2a08941747a] Completed 500 Internal Server Error in 64306ms (ActiveRecord: 2768.4ms | Allocations: 286933)
F, [2022-02-10T18:49:46.982835 #15] FATAL -- : [ebab2068-1248-466e-980c-f2a08941747a]
[ebab2068-1248-466e-980c-f2a08941747a] Parallel::DeadWorker (Parallel::DeadWorker):
[ebab2068-1248-466e-980c-f2a08941747a]
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:78:in `rescue in work'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:75:in `work'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:405:in `block (4 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:518:in `with_instrumentation'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:404:in `block (3 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:392:in `loop'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:392:in `block (2 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:215:in `block (4 levels) in in_threads'

The code in our Rails app that is causing this looks like this (slightly modified for clarity):

ids = ArticleGroupConnection.where(article_group_id: ag_id).pluck(:trip_id)
@all_trips = TripLight.includes(:trip_versions, :trip_dates).order('number')

count_all = ids.size
count_done = 0

group_size = 70

ids.each_slice(group_size) do |group| 
  count_done += group_size 
  trip_share = @all_trips.where(id: group)

   trips_data << Parallel.map(trip_share, in_processes: processcount) do |a_trip|

      Rails.application.executor.wrap do # tried it with and without this, didn't make a difference
        begin
          hash = a_trip.full_hash_v1(false, brand, true) # this is an expensive method with several AR subqueries
          hash
        rescue => err
          Rails.logger.error "######################## ERROR ########################"
          Rails.logger.error  ['api_data', 'each', err.inspect].join(', ')
          Rails.logger.error err.try(:full_message)
          Rails.logger.error err.try(:message)
          Rails.logger.error (err.try(:backtrace) || []).join("\n")
          Rails.logger.error "######################## ERROR END ########################"
          nil
      end
    end
  end
end

When I replace trips_data << Parallel.map(trip_share, in_processes: processcount) do |a_trip| with trips_data << trip_share.map do |a_trip|, everything's fine.

Any idea why this could be happening or how to solve this? Just for reference (and unrelated to parallel), someone else ran into a similar error (but without the stack smashing) after upgrading from Rails 4 to 6: https://github.com/rails/rails/issues/40698

Thanks!

[grosser]

try these https://github.com/grosser/parallel#activerecord if that does not work, then no idea 😞

[jfahrenkrug]

@grosser Thank you for the reply. Maybe it has something to do with the DB adapter we are using in connection with Rails 6 (https://github.com/ibmdb/ruby-ibmdb). What worries me is that it's not simply an error about a missing DB connection, but a low-level memory/stack overflow crash ("stack smashing").

[grosser]

got no idea how this happens, would recommend trying plain threads/forks ... I'd assume it also happens there