Thread puma threadpool terminated with exception machine stack overflow in critical region (fatal)

darrenterhune commented 4 years ago

Issue description

Our CI server is having a problem with the recent major bump to 2.0. Locally we are fine and pdf's work. If I downgrade to 1.4.0 our CI is fine.

Seems this was also an issue in the past: https://stackoverflow.com/questions/55280153/machine-stack-overflow-in-critical-region-fatal https://github.com/mileszs/wicked_pdf/issues/810

Randomized with seed 34635
Capybara starting Puma...
* Version 4.3.3 , codename: Mysterious Traveller
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:34469
#<Thread:0x0000561dc0601600@puma threadpool 001@/home/circleci/project/vendor/bundle/ruby/2.6.0/gems/puma-4.3.3/lib/puma/thread_pool.rb:89 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
/home/circleci/project/vendor/bundle/ruby/2.6.0/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each': machine stack overflow in critical region (fatal)

https://github.com/puma/puma/blob/master/lib/puma/thread_pool.rb#L89 https://github.com/rack/rack/blob/master/lib/rack/urlmap.rb#L58

Expected or desired behavior

CI runs and passes

System specifications

wicked_pdf 2.0.1 wkhtmltopdf-binary-edge 0.12.4.0 because of https://github.com/pallymore/wkhtmltopdf-binary-edge/issues/15 and extra build config just to get it to work so we locked it Ubuntu 16.04 ruby 2.6.5 rails 6.0.2.1

unixmonkey commented 4 years ago

The actual code changes between 1.4.0 and 2.0.1 are pretty small, and I suspect this issue may have to do with a conflict with some other gem that hooks into render, like remotipart (do you have this in your project?).

Please read this comment and look at the diff to see what I'm talking about. I'm curious why this affects CI, but not development. Is CI running the same version of Ruby you develop with? Does it precompile assets, where maybe you don't locally?

darrenterhune commented 4 years ago

Interesting, yeah I didn't see that comment. Let me dive back into this and I'll get back to you. Thanks for some direction 👍

Yes we are using the same ruby locally, on CI and on our production servers. Yes we do have remotipart, it looks like it's a dependency of rails_admin, which we are using.

darrenterhune commented 4 years ago

Ok @unixmonkey I had some more time this avo to debug this. I forked this gem and made this change https://github.com/darrenterhune/wicked_pdf/commit/ccc861596dcd8e43e5243a79b8658cf7c3c9b4e0 and it's now working and passing CI. I just wanted to make sure this comment was true https://github.com/mileszs/wicked_pdf/pull/869/files#r386040878

I also found this where some people are saying having ActiveSupport.on_load ran in multiple gems causes them issues https://github.com/rails/rails/issues/27013 which might be a problem (but I have no idea why) because remotipart also does this https://github.com/JangoSteve/remotipart/blob/master/lib/remotipart/rails/railtie.rb#L33-L36

We just need to 🔪 rails_admin but we are still months off from that and I suspect others will also have this issue.

unixmonkey commented 4 years ago

Yep. Thanks for confirming that. I'm working on a replacement system that avoids any kind of conflict, but I suspect that won't be released for a while either. You might be better off using your fork or monkey-patching the initializer as a no-op, and including/prepending in the controllers you need it in until then.

Jeremie1707 commented 4 years ago

hey, could you elaborate on the way to quick fix this issue : "monkey-patching the initializer as a no-op, and including/prepending in the controllers you need it in until then", while waiting for the next release. i just added rails_admin and I have the same issue as @darrenterhune .

unixmonkey commented 4 years ago

@Jeremie1707 I tried to quickly setup an app to reproduce this, and was not able to. I looked through the commits of rails_admin and see they recently made some updates to improve compatibility with the zeitwerk loader, so maybe it's better now? You could try upgrading rails_admin and remotipart with:

bundle update rails_admin remotipart

And try again.

Another thing that might help and be minimally invasive is to defer requiring wicked_pdf until railties are booted, and your app initializers run. You can do this by editing Gemfile like this:

gem 'wicked_pdf', require: false

Then requiring wicked_pdf before it's needed in the initializer:

require 'wicked_pdf'
WickedPdf.config = {
  exe_path: '/usr/local/bin/wkhtmltopdf'
}

Then only in the controllers it's needed, add the PDF helpers:

class MyController < ApplicationController
  prepend WickedPdf::PdfHelper
  include WickedPdf::WickedPdfHelper::Assets

  def show
  end
end

Or, you can use @darrenterhune 's fork by just changing out your Gemfile

gem 'wicked_pdf', git: 'https://github.com/darrenterhune/wicked_pdf'

Please let me know what you end up doing and what worked or not.

darrenterhune commented 4 years ago

@Jeremie1707 I'd advise against my branch just because I'm not supporting it if there are issues etc. That said it did work for us. However we simply downgraded wicked_pdf for now and locked it so Dependabot stopped trying to upgrade it. The comments about zeitwerk are interesting and I'm going to give this a test today to see what's up because we just in the last couple days had some issues with this new loader stuff. I'll report back after.

darrenterhune commented 4 years ago

Just bumped this gem again. Deferring works but without it, it doesn't work on CI. I have to either use the 1.4 version, defer or use my branch. We've since switched to just deferring since we currently only use this in 1 controller and 1 action in that controller. Adding to application_controller wouldn't be a problem either so probably the better route to go.

andrebras commented 4 years ago

@darrenterhune, @unixmonkey I had the same issue adding the view_component gem (has a monkey patch on the render) and @Okatawa's patch (#919) worked for me.

@darrenterhune can you give it a try? Maybe it's the same issue.

dingzj commented 4 years ago

I got "fatal - exception reentered" error and stuck on loading when showing my rails home page, even the controller/view don't call to generate/display PDF.

Basically, whenever I installed the wicked_pdf gem v2.x above version (just adding to Gemfile, nothing else), the error showed; but, if installed wicked_pdf v 1.4.0 , the error goes away.

My setup: Ruby 2.6.5 Rails 5.2.4.2 wkhtmltopdf-binary: 0.12.6.2 macOS: Mojave I do use: rails_admin (if it matters or not)

After applying @unixmonkey 's comment above, by using the following forked gem, my error goes away.

gem 'wicked_pdf', git: 'https://github.com/darrenterhune/wicked_pdf'

Hope this helps to someone.

rkiller commented 4 years ago

I see the same error:

fatal - machine stack overflow in critical region

Setup: Ruby 2.6.6 Rails 6.0.3.3 wkhtmltopdf-binary 0.12.6.5 macOS: Catalina

I've use version 1.4 of wicked_pdf and no issues at all. Whenever I upgrade to latest this bug shows up and keeps me from upgrading.

feliperaul commented 3 years ago

Does anyone in this thread can reliably reproduce this bug?

I'm also facing it, but only randomly.

Nate Berkopec, a Puma maintainer, is looking into this on https://github.com/puma/puma/issues/2552, but he also can't reproduce the "fatal - machine stack overflow in critical region" error.

Please report in that issue if you can help.

almokhtarbr commented 3 years ago

@feliperaul did you solve it ? i have the same problem !!!

feliperaul commented 3 years ago

@almokhtarbr Nope, and I can't even reproduce it, it's totally random (every once 15 days for us).

For now I have switched to PDFKit, the APIs are very similar.

almokhtarbr commented 3 years ago

@feliperaul the same thing really weird, thanks for suggestion

sposin commented 3 years ago

I am able to reliably get the fatal - machine stack overflow in critical region error.

I encountered it when trying to add Scout APM to our app. scout_apm 4.0.4 ruby 3.0.0 rails 6.0.3.5 wicked_pdf 2.0 wkhtmltopdf-binary (0.12.5.4)

I tried wicked_pdf >= 2.0 and they all have the problem. I downgraded to wicked_pdf 1.4 and the error does not exist.

davegudge commented 3 years ago

I was experiencing :

fatal (exception reentered):

UncaughtThrowError: uncaught throw :app_exception
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:134:in `throw'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:134:in `rescue in call_app'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:131:in `call_app'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:28:in `block in call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `catch'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:26:in `block in call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:80:in `block in tagged'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:28:in `tagged'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:80:in `tagged'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:26:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/sprockets-rails-3.2.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:82:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/webpacker-6.0.0.beta.7/lib/webpacker/dev_server_proxy.rb:25:in `perform_request'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-proxy-0.7.0/lib/rack/proxy.rb:63:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
    from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'

rails (~> 6.0.3.7)
wicked_pdf (2.1.0)
remotipart (1.4.4)

I've added further details to https://github.com/mileszs/wicked_pdf/issues/111#issuecomment-906337853

mathieujobin commented 3 years ago

After encountering the #810 issue.... I upgraded my Rails app from 4.2 to 5.2 while locking down wicked_pdf to 1.1.0 now I am having issues upgrading Rails to 6.1, and I cannot find a working version of wicked_pdf

/home/mathieu/.rvm/rubies/ruby-2.6.8/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each': machine stack overflow in critical region (fatal)

davegudge commented 3 years ago

Have you tried using the master branch of wicked_pdf rather than the latest version?

mathieujobin commented 3 years ago

Issue appears resolved with master branch 5cee578699fe3e5015eb7d76b604a0c121266018

At least CI is happy, I have not test deployed but should be alright.

mathieujobin commented 3 years ago

@davegudge any plan on making a new release ?

davegudge commented 3 years ago

Hey @mathieujobin,

I do not have permission to create a new release, however, @unixmonkey may be able to indicate when a new version is planned for release?

davegudge commented 2 years ago

This is no longer an issue since v2.6.0 was released back in May.

I've just removed this from my Gemfile:

# Tracking master to resolve: https://github.com/mileszs/wicked_pdf/issues/891
gem "wicked_pdf", github: "mileszs/wicked_pdf", branch: "master"

and upgraded to version v2.6.3.

mileszs / wicked_pdf