search

Magickbase / ckb-explorer-public-issues

CKB Explorer Issues
https://explorer.nervos.org/
3 stars 2 forks source link

rufu scheduler error #730

Open zmcNotafraid opened 2 days ago

zmcNotafraid commented 2 days ago 
{ 557060 rufus-scheduler intercepted an error:
  557060   job:
  557060     Rufus::Scheduler::EveryJob "2m" {:overlap=>false}
  557060     ["lib/scheduler.rb", 66]
  557060   error:
  557060     557060
  557060     HTTP::ConnectionError
  557060     failed to connect: getaddrinfo: Temporary failure in name resolution
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `initialize'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `open'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `connect'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/connection.rb:42:in `initialize'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:70:in `new'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:70:in `perform'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/prepend.rb:11:in `block in perform'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/instrumentation.rb:19:in `block in with_tracing'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/tracer.rb:355:in `capture_segment_error'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/instrumentation.rb:19:in `with_tracing'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/prepend.rb:11:in `perform'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:31:in `request'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/chainable.rb:75:in `request'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/chainable.rb:27:in `post'
  557060       /usr/src/app/models/ckb_sync/api.rb:159:in `directly_single_call_rpc'
  557060       /usr/src/app/workers/pool_transaction_check_worker.rb:10:in `block in perform'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation/delegation.rb:88:in `each'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation/delegation.rb:88:in `each'
  557060       /usr/src/app/workers/pool_transaction_check_worker.rb:9:in `perform'
  557060       lib/scheduler.rb:29:in `call_worker'
  557060       lib/scheduler.rb:67:in `block in <main>'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:233:in `block in do_call'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `cache'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/query_cache.rb:11:in `cache'
  557060       lib/scheduler.rb:19:in `block (2 levels) in around_trigger'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
  557060       lib/scheduler.rb:18:in `block in around_trigger'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
  557060       lib/scheduler.rb:17:in `around_trigger'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:232:in `do_call'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:280:in `trigger_now'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:331:in `block (3 levels) in start_work_thread'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:334:in `block (2 levels) in start_work_thread'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:317:in `loop'
  557060       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:317:in `block in start_work_thread'
  557060   tz:
  557060     ENV['TZ']: 
  557060     Time.now: 2024-07-03 02:34:21 +0000
  557060     local_tzone: #<TZInfo::DataTimezone: Asia/Shanghai>
  557060   et-orbi:
  557060     (etz:nil,tnz:"UTC",tziv:"2.0.6",tzidv:nil,rv:"3.1.2",rp:"x86_64-linux",win:false,rorv:"7.0.8.4",astz:[ActiveSupport::TimeZone, "Asia/Shanghai"],eov:"1.2.7",eotnz:#<TZInfo::DataTimezone: Asia/Shanghai>,eotnfz:"+0800",eotlzn:"Asia/Shanghai",eotnfZ:"CST",debian:"Etc/UTC",centos:nil,osx:"Etc/UTC")
  557060   scheduler:
  557060     object_id: 47900
  557060     opts:
  557060       {}
  557060       frequency: 0.3
  557060       scheduler_lock: #<Rufus::Scheduler::NullLock:0x00007efd02f948f8>
  557060       trigger_lock: #<Rufus::Scheduler::NullLock:0x00007efd02f94880>
  557060     uptime: 2538.0625336170197 (42m18.062560797s)
  557060     down?: false
  557060     frequency: 0.3
  557060     discard_past: true
  557060     started_at: #<EtOrbi::EoTime:0x00007efd02f94420 @seconds=1719971523.273509, @zone=#<TZInfo::DataTimezone: Asia/Shanghai>, @time=nil>
  557060     paused_at: nil
  557060     threads: 6
  557060       thread: #<Thread:0x00007efd02f9bb58 /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler.rb:634 sleep>
  557060       thread_key: rufus_scheduler_47900
  557060       work_threads: 5
  557060         active: 1
  557060         vacant: 4
  557060         max_work_threads: 28
  557060       mutexes: {}
  557060     jobs: 23
  557060       at_jobs: 0
  557060       in_jobs: 0
  557060       every_jobs: 14
  557060       interval_jobs: 3
  557060       cron_jobs: 6
  557060     running_jobs: 1
  557060     work_queue:
  557060       size: 0
  557060       num_waiting: 4
  557060     join_queue:
  557060       size: 0
  557060       num_waiting: 1
} 557060 .
{ 603420 rufus-scheduler intercepted an error:
  603420   job:
  603420     Rufus::Scheduler::EveryJob "2m" {:overlap=>false}
  603420     ["lib/scheduler.rb", 66]
  603420   error:
  603420     603420
  603420     HTTP::ConnectionError
  603420     failed to connect: getaddrinfo: Temporary failure in name resolution
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `initialize'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `open'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/timeout/null.rb:21:in `connect'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/connection.rb:42:in `initialize'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:70:in `new'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:70:in `perform'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/prepend.rb:11:in `block in perform'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/instrumentation.rb:19:in `block in with_tracing'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/tracer.rb:355:in `capture_segment_error'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/instrumentation.rb:19:in `with_tracing'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.12.0/lib/new_relic/agent/instrumentation/httprb/prepend.rb:11:in `perform'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/client.rb:31:in `request'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/chainable.rb:75:in `request'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/http-5.1.0/lib/http/chainable.rb:27:in `post'
  603420       /usr/src/app/models/ckb_sync/api.rb:159:in `directly_single_call_rpc'
  603420       /usr/src/app/workers/pool_transaction_check_worker.rb:10:in `block in perform'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation/delegation.rb:88:in `each'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation/delegation.rb:88:in `each'
  603420       /usr/src/app/workers/pool_transaction_check_worker.rb:9:in `perform'
  603420       lib/scheduler.rb:29:in `call_worker'
  603420       lib/scheduler.rb:67:in `block in <main>'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:233:in `block in do_call'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `cache'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/query_cache.rb:11:in `cache'
  603420       lib/scheduler.rb:19:in `block (2 levels) in around_trigger'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
  603420       lib/scheduler.rb:18:in `block in around_trigger'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
  603420       lib/scheduler.rb:17:in `around_trigger'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:232:in `do_call'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:280:in `trigger_now'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:331:in `block (3 levels) in start_work_thread'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:334:in `block (2 levels) in start_work_thread'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:317:in `loop'
  603420       /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler/jobs_core.rb:317:in `block in start_work_thread'
  603420   tz:
  603420     ENV['TZ']: 
  603420     Time.now: 2024-07-03 02:38:14 +0000
  603420     local_tzone: #<TZInfo::DataTimezone: Asia/Shanghai>
  603420   et-orbi:
  603420     (etz:nil,tnz:"UTC",tziv:"2.0.6",tzidv:nil,rv:"3.1.2",rp:"x86_64-linux",win:false,rorv:"7.0.8.4",astz:[ActiveSupport::TimeZone, "Asia/Shanghai"],eov:"1.2.7",eotnz:#<TZInfo::DataTimezone: Asia/Shanghai>,eotnfz:"+0800",eotlzn:"Asia/Shanghai",eotnfZ:"CST",debian:"Etc/UTC",centos:nil,osx:"Etc/UTC")
  603420   scheduler:
  603420     object_id: 47900
  603420     opts:
  603420       {}
  603420       frequency: 0.3
  603420       scheduler_lock: #<Rufus::Scheduler::NullLock:0x00007efd02f948f8>
  603420       trigger_lock: #<Rufus::Scheduler::NullLock:0x00007efd02f94880>
  603420     uptime: 2771.6267042160034 (46m11.626728296s)
  603420     down?: false
  603420     frequency: 0.3
  603420     discard_past: true
  603420     started_at: #<EtOrbi::EoTime:0x00007efd02f94420 @seconds=1719971523.273509, @zone=#<TZInfo::DataTimezone: Asia/Shanghai>, @time=nil>
  603420     paused_at: nil
  603420     threads: 6
  603420       thread: #<Thread:0x00007efd02f9bb58 /usr/src/vendor/bundle/ruby/3.1.0/gems/rufus-scheduler-3.8.2/lib/rufus/scheduler.rb:634 sleep>
  603420       thread_key: rufus_scheduler_47900
  603420       work_threads: 5
  603420         active: 2
  603420         vacant: 3
  603420         max_work_threads: 28
  603420       mutexes: {}
  603420     jobs: 23
  603420       at_jobs: 0
  603420       in_jobs: 0
  603420       every_jobs: 14
  603420       interval_jobs: 3
  603420       cron_jobs: 6
  603420     running_jobs: 2
  603420     work_queue:
  603420       size: 0
  603420       num_waiting: 3
  603420     join_queue:
  603420       size: 0
  603420       num_waiting: 1
} 603420 .
zmcNotafraid commented 2 days ago

It was caused by ckb node.Ops has adjust the node, we will watch them few days. PR: https://github.com/nervosnetwork/ckb-explorer/pull/2018 https://github.com/nervosnetwork/ckb-explorer/pull/2027