search

zombocom / rack-timeout

Abort requests that are taking too long
MIT License
878 stars 156 forks source link

undefined method `state=' for nil:NilClass since 0.3.1 #92

Closed nvloff closed 6 years ago

nvloff commented 9 years ago

Yesterday we've upgraded rack-timeout to 0.3.1 from 0.2.4 and we started seeing this error on production (we're using heroku) :


    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/core.rb:168 :in `_set_state!`

    166     def self._set_state!(env, state)
    167       raise "Invalid state: #{state.inspect}" unless VALID_STATES.include? state
    168       env[ENV_INFO_KEY].state = state
    169       notify_state_change_observers(env)
    170     end

    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/core.rb:115 :in `block in call`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/core.rb:117 :in `call`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/core.rb:117 :in `block in call`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:39 :in `call`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:39 :in `run!`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:52 :in `run!`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:99 :in `block (2 levels) in run_loop!`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:99 :in `each`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:99 :in `block in run_loop!`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:80 :in `loop`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:80 :in `run_loop!`
    [GEM_ROOT]/gems/rack-timeout-0.3.1/lib/rack/timeout/support/scheduler.rb:72 :in `block (2 levels) in runner`

Rack middlewares on production:

use Honeybadger::Rack::UserInformer
use Honeybadger::Rack::UserFeedback
use Honeybadger::Rack::ErrorNotifier
use HireFire::Middleware
use Rack::Cors
use Rack::UTF8Sanitizer
use Rack::Sendfile
use ActionDispatch::Static
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x007f7d7d3117b0>
use Rack::Timeout
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::RemoteIp
use ActionDispatch::Callbacks
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Warden::Manager
use JQuery::FileUpload::Rails::Middleware

Tried reproducing locally but I can't seem to find the right conditions to do so.

kch commented 8 years ago

@DSAMPAT heroku has no issues with git branch gems. I even deployed a test app to confirm. What are you seeing that makes you believe that? Is it failing to deploy? Failing to start? Paste any relevant logs here.

DSAMPAT commented 8 years ago

@kch I have managed to deploy. Not sure what the issue was. Could have been another gem causing the bundle issue as I am using Windows mingw-32.

I am getting this notification below, should I change anything or is this fine? app/web.1:Rack::Timeout.timeout=: class-level settings are deprecated. See README for examples on using the middleware initializer instead.

kch commented 8 years ago

@DSAMPAT it's just a warning, doesn't really impact anything at the moment.

beatjoerg commented 8 years ago

We are seeing this issue also. I will try with your debug branch.

beatjoerg commented 8 years ago

Unfortunately we are on ruby 2.0... so the beta version is not going to work. Can you make a version of the 0.3.2?

kch commented 8 years ago

@beatjoerg it should work fine with 2.0. Any reason you think it wouldn't?

beatjoerg commented 8 years ago

@kch Because you use some methods in the code that are not available with ruby 2.0, like super_method. I tried to backport them, but I did not arrive to fix all in the timebox I had. To see what I "fixed" so far: https://github.com/beatjoerg/rack-timeout/commits/26e05ee40322eae527aa3fd38a8d933e3dec1819/lib/rack/timeout/legacy.rb

kch commented 8 years ago

Oh super_method is really bad, it's 2.2-only. Ok, I'll get rid of that.

kch commented 8 years ago

@beatjoerg 0.4.2 is out with 2.0 compatibility. I also rebased the debug branch here so it should be good to go now.

@samandmoore might work for you in jruby now too, give it a go.

samandmoore commented 8 years ago

@kch thanks. Still no luck for me running it because it's not ruby 1.9 compatible:

SyntaxError: /opt/rubies/jruby-1.7.24/lib/ruby/gems/shared/bundler/gems/rack-timeout-7a5914cc9734/lib/rack/timeout/core.rb:94: syntax error, unexpected tLABEL
    def initialize(app, service_timeout:nil, wait_timeout:nil, wait_overtime:nil, service_past_wait:false)
kch commented 8 years ago

@samandmoore ok so your jruby version also doesn't support named parameters. I'll see if I can make a version of the debug branch off 0.3.2…

kch commented 8 years ago

@samandmoore give it a go https://github.com/heroku/rack-timeout/tree/vanishing-env-info-debug-0.3.2

samandmoore commented 8 years ago

thanks, will give it a shot.

kch commented 8 years ago

Is anyone actually running this debug branch in a reasonably trafficked app? I'd've expected to have some useful log lines by now.

How about this: I'll mail some heroku swag to the first person who delivers log lines that enlighten me. Help us if you like purple stuff!

samandmoore commented 8 years ago

I'm working on it. Just in the midst of another big change. Will likely have results for you tomorrow :) On Wed, Apr 6, 2016 at 18:25 Caio Chassot notifications@github.com wrote:

Is anyone actually running this debug branch in a reasonably trafficked app? I'd've expected to have some useful log lines by now.

How about this: I'll mail some heroku swag to the first person who delivers log lines that enlighten me. Help us if you like purple stuff!

— You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub https://github.com/heroku/rack-timeout/issues/92#issuecomment-206595820

beatjoerg commented 8 years ago

@kch Just deployed to production... Waiting for the errors...

mbasset commented 8 years ago

@kch Unfortunately last time we tried this branch we almost took down our production. We have created a bypass for the error but it still happens at quite a regular frequency. ~200 per hour.

kch commented 8 years ago

@mbasset I wonder if only injecting the hijack on a sample (like 10% or so) of the requests would make it more bearable for you. Lmk if it sounds like a good a idea and I'll add that as an option.

mbasset commented 8 years ago

@kch That sounds good in theory but even having it on 10% of our traffic could cause our servers to become unstable and die, albeit more slowly. We may be able to try this in our staging environment as we recently created a script to simulate some production load. However need to find time to try this out.

kch commented 8 years ago

@mbasset well I'd make the frequency configurable. Also I believe I rewrote the debug code to be a lot nicer since you last tried it. Not sure.

kch commented 8 years ago

@mbasset so I did it. You can set the env var RACK_TIMEOUT_HIJACK_FREQUENCY to any number between 0 and 1.0 and that'll determine the frequency.

I've also made changes so I'm hijacking even less methods, trying to stay more out of the way.

mbasset commented 8 years ago

Cool I'll see when we get a chance to try this version out and see what happens.

mbasset commented 8 years ago

Just FYI still have not had a chance to look into this yet. Ideally if someone else has time.

mbasset commented 8 years ago

@kch Hoping to try out the branch next week. However I see it only sends warnings. In our production environment (which is the only environment this seems to happen in) how would you suggest getting the output. We are currently using an nginx/unicorn setup.

samandmoore commented 8 years ago

I've been working on updating the 0.3.2 branch to look like the newer one. I tried to run it under a load test (because we only see these issues when under reasonable load) and I haven't been able to reproduce it in development mode. However, I was able to reproduce it in a capybara feature spec. Unfortunately, that spec didn't have the logging in place. I am hopefully going to try it in production in the next week or so.

On Fri, Apr 29, 2016 at 3:17 PM Matthew Basset notifications@github.com wrote:

@kch https://github.com/kch Hoping to try out the branch next week. However I see it only sends warnings. In our production environment (which is the only environment this seems to happen in) how would you suggest getting the output. We are currently using an nginx/unicorn setup.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/heroku/rack-timeout/issues/92#issuecomment-215853100

mbasset commented 8 years ago

@kch We have been trying your latest branch out in an environment that normally causes ~100 of these errors an hour with rails 4.2.6 and it does not appear to be happening anymore. I wonder if the changes that were made to timing in that branch have solved the issue. Possibly also fixed with rails update. Will continue to monitor for the next while to see.

beatjoerg commented 8 years ago

@kch @mbasset : Same also here, we do not see the error anymore running the debug branch. (We use also rails 4.2.6.)

samandmoore commented 8 years ago

@kch I have a version of the debug branch in production, and i'm seeing occasional log statements like this:

[DM4PNP]2016-05-24 07:15:53 -0400 severity=WARN, source=rack-timeout-debug info-vanish id=4773927afb3ba69e09eed01d8ae1b6fd

Unfortunately, there are no other logs associated with that id value, so I've got no call stack for you. That makes me think that the method that is unsetting the env info is one of the methods we're not redefining with the logging behavior. Thoughts?

nadavshatz commented 8 years ago

I'm sorry for bumping this for everyone - did anyone have any success in resolving this?

mbasset commented 8 years ago

Rails 4.2.6 and the latest 0.4.2 version has fixed it entirely for us.

nextofsearch commented 8 years ago

Hi, I am suffering from the same issue with rack-timeout 0.3.2, ruby 1.9.3-p551 and Rails 3.2.22.5. Unfortunately I am stuck with what I have (can't upgrade ruby or Rails.). It seems that ruby 1.9.x works with 0.3.x so I can't use 0.4.x. I will try downgrade to 0.2.x and keep you posted.

wuputah commented 6 years ago

It appears this has been resolved in some way or otherwise gone away, so closing pending new reports.

katrinleinweber commented 4 years ago

I saw essentially the same error in v0.5.1 (slightly different line numbers) and with Rails v6.0.3 and gitlab-puma v4.3.3.gitlab.2. It was preceded by 

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-timeout-0.5.1/lib/rack/timeout/support/scheduler.rb:73 run> terminated with exception (report_on_exception is true):
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-timeout-0.5.1/lib/rack/timeout/core.rb:168:in `_set_state!': undefined method `state=' for nil:NilClass (NoMethodError)

The only (remote) clue I have to hopefully help reproduce the issue, is that Unicode characters Ç & Ã are in a commit message, that's not being displayed where it should be. I'm grasping at straws here, but is string parsing involved at any point here?

vermeeren commented 4 years ago

Have been having this issue since upgrade to GitLab 13.3.2-ee (13.2.x did not happen). This is with 0.5.2, which I realise is not the latest version of this package. The two instances with EE Premium licence have this problem and a service restart "fixes" it until it occurs again. Another instance without a licence uploaded appears to not have this problem, but it could just be RNG. I will raise an issue with GitLab support regarding this.

``` # terminated with exception (report_on_exception is true): /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:168:in `_set_state!': undefined method `state=' for nil:NilClass (NoMethodError) from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:113:in `block in call' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:115:in `block in call' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:39:in `run!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:52:in `run!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `block (2 levels) in run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `block in run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:81:in `loop' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:81:in `run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:73:in `block (2 levels) in runner' /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:168:in `_set_state!': undefined method `state=' for nil:NilClass (NoMethodError) from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:113:in `block in call' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:115:in `block in call' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:39:in `run!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:52:in `run!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `block (2 levels) in run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:100:in `block in run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:81:in `loop' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:81:in `run_loop!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:73:in `block (2 levels) in runner' ```
vermeeren commented 4 years ago

On another host the traceback is very different, so it is almost certain it is a problem in Puma or some component of GitLab and not this project, apologies for the noise. It also happened with GitLab 13.2.x fyi.

``` var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:837:in `block in with_new_connections_blocked': undefined method `clear' for nil:NilClass (NoMethodError) from /usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:833:in `ensure in with_new_connections_blocked' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:850:in `with_new_connections_blocked' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:752:in `with_exclusively_acquired_all_connections' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:503:in `disconnect' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:525:in `disconnect!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1136:in `remove_connection' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1054:in `establish_connection' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:51:in `establish_connection' from /var/opt/gitlab/gitlab/config/initializers/active_record_lifecycle.rb:8:in `block (2 levels) in ' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:71:in `class_eval' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:42:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:42:in `on_load' from /var/opt/gitlab/gitlab/config/initializers/active_record_lifecycle.rb:7:in `block in ' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:147:in `each' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:147:in `call' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:114:in `do_worker_start' from /var/opt/gitlab/gitlab/config/puma.rb:57:in `block in _load_from' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `block in run_hooks' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `run_hooks' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:267:in `worker' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:138:in `block (2 levels) in spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:138:in `block in spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:134:in `times' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:134:in `spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:204:in `check_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:487:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/launcher.rb:172:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cli.rb:80:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/bin/puma:10:in `' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/puma:23:in `load' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/puma:23:in `
' /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:761:in `block in attempt_to_checkout_all_existing_connections': private method `select' called for nil:NilClass (NoMethodError) from /usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:759:in `attempt_to_checkout_all_existing_connections' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:753:in `block in with_exclusively_acquired_all_connections' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:829:in `with_new_connections_blocked' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:752:in `with_exclusively_acquired_all_connections' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:503:in `disconnect' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:525:in `disconnect!' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1136:in `remove_connection' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1054:in `establish_connection' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:51:in `establish_connection' from /var/opt/gitlab/gitlab/config/initializers/active_record_lifecycle.rb:8:in `block (2 levels) in ' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:71:in `class_eval' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:42:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:42:in `on_load' from /var/opt/gitlab/gitlab/config/initializers/active_record_lifecycle.rb:7:in `block in ' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:147:in `each' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:147:in `call' from /var/opt/gitlab/gitlab/lib/gitlab/cluster/lifecycle_events.rb:114:in `do_worker_start' from /var/opt/gitlab/gitlab/config/puma.rb:57:in `block in _load_from' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `block in run_hooks' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `each' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:279:in `run_hooks' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:267:in `worker' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:138:in `block (2 levels) in spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:138:in `block in spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:134:in `times' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:134:in `spawn_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:204:in `check_workers' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cluster.rb:487:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/launcher.rb:172:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cli.rb:80:in `run' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-puma-4.3.3.gitlab.2/bin/puma:10:in `' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/puma:23:in `load' from /var/opt/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/puma:23:in `
' ```