instacart / makara

A Read-Write Proxy for Connections; Also provides an ActiveRecord adapter.
http://tech.taskrabbit.com/
MIT License
926 stars 171 forks source link

undefined method `query_cache_enabled' for nil:NilClass #182

Open palexvs opened 6 years ago

palexvs commented 6 years ago

I get an exception undefined method 'query_cache_enabled' for nil:NilClass when master recovers after downtime for the first requests. There is a backtrace:

 ".../activerecord-5.1.4/lib/active_record/connection_adapters/abstract/query_cache.rb:130:in `configure_query_cache!'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `block in make_lambda'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:270:in `block in simple'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `block in invoke_after'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `each'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `invoke_after'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:132:in `run_callbacks'",
 ".../activesupport-5.1.4/lib/active_support/callbacks.rb:827:in `_run_checkout_callbacks'",
 ".../makara-a3c83e14412d/lib/makara/connection_wrapper.rb:99:in `public_send'",
 ".../makara-a3c83e14412d/lib/makara/connection_wrapper.rb:99:in `method_missing'",
 ".../makara-a3c83e14412d/lib/makara/proxy.rb:106:in `public_send'",
 ".../makara-a3c83e14412d/lib/makara/proxy.rb:106:in `block in method_missing'",
 ".../makara-a3c83e14412d/lib/makara/proxy.rb:153:in `block in any_connection'",
 ".../makara-a3c83e14412d/lib/makara/pool.rb:98:in `block in provide'",
 ".../makara-a3c83e14412d/lib/active_record/connection_adapters/makara_abstract_adapter.rb:37:in `handle'",
 ".../makara-a3c83e14412d/lib/makara/pool.rb:97:in `provide'",
 ".../makara-a3c83e14412d/lib/makara/proxy.rb:152:in `any_connection'",
 ".../makara-a3c83e14412d/lib/makara/proxy.rb:104:in `method_missing'",
 ".../activerecord-5.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:807:in `checkout_and_verify'",
 ".../activerecord-5.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:500:in `checkout'",
 ".../activerecord-5.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:374:in `connection'",
 ".../activerecord-5.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:931:in `retrieve_connection'",
 ".../activerecord-5.1.4/lib/active_record/connection_handling.rb:116:in `retrieve_connection'",
 ".../activerecord-5.1.4/lib/active_record/connection_handling.rb:88:in `connection'",
 ".../activerecord-5.1.4/lib/active_record/relation/delegation.rb:44:in `connection'",
 ".../activerecord-5.1.4/lib/active_record/relation/query_methods.rb:81:in `bound_attributes'",
 ".../activerecord-5.1.4/lib/active_record/relation.rb:667:in `exec_queries'",
 ".../activerecord-5.1.4/lib/active_record/relation.rb:546:in `load'",
 ".../activerecord-5.1.4/lib/active_record/relation.rb:255:in `records'",
 ".../activerecord-5.1.4/lib/active_record/relation.rb:251:in `to_a'",
 ".../activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:546:in `find_nth_with_limit'",
 ".../activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:531:in `find_nth'",
 ".../activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:122:in `first'",
 ".../my_project.rb:24:in `find_by_uuid'"

database.yml:

base: &base
  adapter: mysql2_makara
  encoding: utf8mb4
  collation: utf8mb4_unicode_ci
  reconnect: true
  pool: 15
  username: <%= SENV['MYSQL_USER'] %>
  password: <%= SENV['MYSQL_PASSWORD'] %>
  database: <%= SENV['MYSQL_DATABASE'] %>
  variables: # https://github.com/rails/rails/issues/25924
    sql_mode: TRADITIONAL

  makara:
    sticky: false
    master_strategy: failover
    slave_strategy: failover
    connection_error_matchers:
      - '/Server shutdown in progress/' # Do not fail on reboot
      - '/Unknown MySQL server host/' # Can be thrown on replica re-creation
    connect_timeout: 1

    connections:
      - role: master
        host: master.host
        port: 3307
        sticky: true
        disable_blacklist: true
      - role: slave
        host: slave.host
        port: 3308
        blacklist_duration: 60
wpdavenport commented 6 years ago

I am also seeing this same issue on Rails 5.0.6. However, I'm not seeing it on Rails 4.2.8 using the same configuration on both apps.

What I have noticed is that this error can happen from one request to the next. Initially, I thought it may be due to a session timeout, but I found that it can happen on any type request (i.e. GET, POST). However, subsequent refresh's of the same page or action (like a remove JS call) and the error will go away for while.

Is there something else that needs to be set to resolve this?

Error

NoMethodError: undefined method `query_cache_enabled' for nil:NilClass
  from active_record/connection_adapters/abstract/query_cache.rb:120:in `configure_query_cache!'
  from active_support/callbacks.rb:382:in `block in make_lambda'
  from active_support/callbacks.rb:242:in `block in simple'
  from active_support/callbacks.rb:456:in `block in call'
  from active_support/callbacks.rb:456:in `each'
  from active_support/callbacks.rb:456:in `call'
  from active_support/callbacks.rb:101:in `__run_callbacks__'
  from active_support/callbacks.rb:750:in `_run_checkout_callbacks'
  from makara/connection_wrapper.rb:99:in `public_send'
  from makara/connection_wrapper.rb:99:in `method_missing'
  from makara/proxy.rb:106:in `public_send'
  from makara/proxy.rb:106:in `block in method_missing'
  from makara/proxy.rb:153:in `block in any_connection'
  from makara/pool.rb:105:in `block in provide'
  from active_record/connection_adapters/makara_abstract_adapter.rb:37:in `handle'
  from makara/pool.rb:104:in `provide'
  from makara/proxy.rb:152:in `any_connection'
  from makara/proxy.rb:104:in `method_missing'
  from active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify'
  from active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout'
  from active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection'
  from active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection'
  from active_record/connection_handling.rb:128:in `retrieve_connection'
  from active_record/connection_handling.rb:91:in `connection'
  from active_record/relation/delegation.rb:43:in `connection'
  from active_record/relation/query_methods.rb:105:in `bound_attributes'
  from active_record/relation.rb:706:in `exec_queries'
  from active_record/relation.rb:583:in `load'
  from active_record/relation.rb:260:in `records'
  from active_record/relation.rb:256:in `to_a'
  from active_record/relation/finder_methods.rb:563:in `find_nth_with_limit'
  from active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset'
  from active_record/relation/finder_methods.rb:545:in `find_nth'
  from active_record/relation/finder_methods.rb:122:in `first'
  from orm_adapter/adapters/active_record.rb:17:in `get'
  from devise/models/authenticatable.rb:229:in `serialize_from_session'
  from devise.rb:478:in `block (2 levels) in configure_warden!'
  from warden/session_serializer.rb:35:in `fetch'

database.yml

staging:
  adapter:  postgresql_makara
  encoding: unicode
  pool:     10

  makara:
    blacklist_duration: 0
    disable_blacklist: true
    master_ttl: 5
    master_strategy: round_robin
    sticky: true

    connections:
      - role: master
        url: <removed>
      - role: slave
        url: <removed>
arpitvaishnav commented 6 years ago

I am having similar error like @palexvs , while setting up aws aurora db with master and slave configuration. My Stacks are

SVMKwangillJung commented 6 years ago

I had a same error like @palexvs and @wpdavenport.

by uninstalling out all the gems and install again, the error is gone for now.

using this command

gem uninstall -aIx

arpitvaishnav commented 5 years ago

I am using docker container sytem, so every time any change comes on Dockerfile, docker runs bundle install for fresh installation. I dont think gem uninstall -aIx will solve my issue. Is there any solution to this issue? Any reason behind this error? So that we can resolve this ? After updating gem to 0.4.0 I am facing issue explained by @wpdavenport .

tmedford commented 5 years ago

Anybody solve this?

wendy0402 commented 4 years ago

this issue start happens on me as well when I disable the blacklist feature

lukluk commented 4 years ago

me too, using distribute_reads (0.2.4)