linqueta / rails-healthcheck

A simple way to configure a healthcheck route for a Rails application
MIT License
136 stars 10 forks source link

Strange issues with the threading #44

Closed jcavalieri closed 3 years ago

jcavalieri commented 3 years ago

I'm having a strange issue related to the the threaded calls of configured checks.

I have not been able to tell what is going on with putting in more logging or changing how errors are being handled. This only happens with actual requests, in console it works fine. The only thing that works for me right now is to monkey patch and remove the threading.

# Monkey Patch to remove threading from the rails-healthcheck gem
module Healthcheck
  class Checker
    def check
      Healthcheck.configuration.checks.map { |c| execute(c) }
    end
  end
end

Since it is only happening via request, it makes me think it is Puma related.

Here are some details:

web_1  | 2021-01-10T01:36:54.647033800Z => Booting Puma
web_1  | 2021-01-10T01:36:54.648270500Z => Rails 5.2.3 application starting in development
web_1  | 2021-01-10T01:36:54.648314800Z => Run `rails server -h` for more startup options
web_1  | 2021-01-10T01:36:54.648331700Z Puma starting in single mode...
web_1  | 2021-01-10T01:36:54.648344100Z * Puma version: 5.1.1 (ruby 2.5.8-p224) ("At Your Service")
web_1  | 2021-01-10T01:36:54.648359200Z *  Min threads: 5
web_1  | 2021-01-10T01:36:54.648373600Z *  Max threads: 5
web_1  | 2021-01-10T01:36:54.648393100Z *  Environment: development
web_1  | 2021-01-10T01:36:54.648407100Z *          PID: 9
web_1  | 2021-01-10T01:36:54.648436800Z * Listening on http://0.0.0.0:3000

Any ideas?

jcavalieri commented 3 years ago

Hmm, maybe this is related: https://guides.rubyonrails.org/threading_and_code_execution.html

jcavalieri commented 3 years ago

Just a follow up. So if I run under RACK_ENV=production everything works fine. It is just under RACK_ENV=development where there is an issue. It does appear to be an eager-load vs auto-load issue. I haven't solved it but I did modify the monkey patch for now.

module Healthcheck
  # Monkey Patch to remove threading from the rails-healthcheck gem
  class Checker
    def check
      if Rails.env.development?
        # You can't do multi-threading with requests in development mode due to auto-load vs eager-load
        # TODO: supposedly this problem goes away with Rails 6
        Healthcheck.configuration.checks.map { |c| execute(c) }
      else
        Healthcheck.configuration
                   .checks
                   .map { |c| Thread.new { execute(c) } }
                   .each(&:join)
      end
    end
  end
end

Also, I believe this problem goes away with Rails 6.

linqueta commented 3 years ago

Hey man, are you ok?

So, I'll test this scenario and come back here to tell what I understood about this point.

jcavalieri commented 3 years ago

With the monkey patch I've got it working. I'm struggling to pinpoint exactly what is going on. Nothing gets logged. I think there is something regarding the loading of classes/constants within the thread. And that is related to how Rails auto-loads or eager-loads.

I really think this is a Rails issue, not an issue with this Gem.

Just an idea, but what if we made the threading optional/configurable? I'm assuming I'm not the only one that will run into this issue.

jcavalieri commented 3 years ago

Well, I went through the whole upgrade process to rails 6, and that didn't solve the issue.

However, setting eager_load to true in config/environments/development.rb did seem to solve the issue

 # Do not eager load code on boot.
 config.eager_load = true

~But note, setting config.eager_load = true did not work for Rails 5.2.~

Correction: Setting config.eager_load = true in config/environments/development.rb does work in Rails 5.2.