search

instructure / canvas-lms

The open LMS by Instructure, Inc.
https://github.com/instructure/canvas-lms/wiki
GNU Affero General Public License v3.0
5.6k stars 2.48k forks source link

Redis errors in a production log. #1771

Open amg-web opened 3 years ago

amg-web commented 3 years ago

Summary:

Fresh installed Canvas in production environment Ubuntu 18.04 Redis 6.0.6 Ruby 2.6 have to add some extra packages:

cache_store.yml

production:
    cache_store: redis_cache_store

redis.yml

production:
        servers:
                -  redis://localhost

production.log - errors

# Logfile created on 2020-12-08 10:25:00 +0000 by logger.rb/66358
[- -] Creating scope :active. Overwriting existing method AssignmentGroup.active.
[- -] Creating scope :available_to_planner. Overwriting existing method Announcement.available_to_planner.
[- -] Creating scope :open. Overwriting existing method GradingPeriod.open.
[- -]   [REDIS] Query failure #<Redis::InheritedError: Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.> (localhost:6379)
[- -] Failure handling redis command on localhost:6379: #<Redis::InheritedError: Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.>
[- -]

[CANVAS_ERRORS] EXCEPTION LOG
Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.):
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:375:in `ensure_connected'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
  /home/canvas/public_html/lib/canvas/redis.rb:180:in `block in process'
  /home/canvas/public_html/lib/canvas/redis.rb:96:in `handle_redis_failure'
  /home/canvas/public_html/lib/canvas/redis.rb:179:in `process'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:914:in `block in get'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize'
  /usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:913:in `get'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/redis_cache_store.rb:324:in `block (2 levels) in read_entry'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/redis_cache_store.rb:26:in `with'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/redis_cache_store.rb:324:in `block in read_entry'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/redis_cache_store.rb:451:in `failsafe'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/redis_cache_store.rb:322:in `read_entry'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache/strategy/local_cache.rb:126:in `read_entry'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache.rb:313:in `block in fetch'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache.rb:663:in `block in instrument'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/notifications.rb:168:in `block in instrument'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/notifications.rb:168:in `instrument'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache.rb:663:in `instrument'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.4/lib/active_support/cache.rb:312:in `fetch'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.2/app/models/switchman/shard.rb:571:in `find_cached'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.2/app/models/switchman/shard.rb:66:in `default'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.2/lib/switchman/active_record/connection_handler.rb:85:in `establish_connection'
  /home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.4/lib/active_record/connection_handling.rb:60:in `establish_connection'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:321:in `before_handling_requests'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:159:in `block in negotiate_spawn_command'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:382:in `run_block_and_record_step_progress'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `negotiate_spawn_command'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:211:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'CONTEXT: {:tags=>{:type=>"redis"}}

[- -]   [REDIS] Short circuiting due to recent redis failure (localhost:6379)
[fc3e613a147ba24245f8b8332ef1e38d 16c3a6f7-1eb4-40f3-b52c-12f1d64a9b74]   [REDIS] Short circuiting due to recent redis failure (localhost:6379)

[fc3e613a147ba24245f8b8332ef1e38d 088436f1-4aea-4f97-9c1a-ece352f04c9a]   [REDIS] Query failure #<Redis::CommandError: NOSCRIPT No matching script. Please use EVAL.> (localhost:6379)
[fc3e613a147ba24245f8b8332ef1e38d 01f9c3d9-a7b3-4914-a9d1-99db67c4a1c6] No separate files host specified for account id 1.  This is a potential security risk.
shawngsli commented 3 years ago

We encountered this issue too and found out it is caused by passenger 6.x.

If you are using passenger 6.x, try downgrade to 5.x and see if it can be solved.

amg-web commented 3 years ago

We encountered this issue too and found out it is caused by passenger 6.x.

If you are using passenger 6.x, try downgrade to 5.x and see if it can be solved.

Redis gem updated to 4.1.4 in stable. After this update no errors even with passenger 6.x