search

openSUSE / open-build-service

Build and distribute Linux packages from sources in an automatic, consistent and reproducible way #obs
https://openbuildservice.org
GNU General Public License v2.0
934 stars 438 forks source link

obsapidelayed is eating away my CPU #830

Closed jengelh closed 9 years ago

jengelh commented 9 years ago

Updated from obs-api-2.5.5 to obs-api-2.5.93 this morning (because I want the "anonymous" mode fix).

After the update, I try to restart services. And I find them overly busy. The api_production.delayed_jobs table has over 120000 entries, the "delayed_job.1010" and "delayed_job.1020" processes are busy working (as is mysqld), blocking clients from successfully contacting the API. There is a TON of entries in the table like

MariaDB [api_production]> SELECT * FROM delayed_jobs LIMIT 1;
*************************** 1. row ***************************
        id: 1457
  priority: 0
  attempts: 1
   handler: --- !ruby/object:Delayed::PerformableMethod
object: !ruby/ActiveRecord:IssueTracker
  attributes:
    id: 15
    name: bnc
    kind: bugzilla
    description: openSUSE Bugzilla
    url: https://bugzilla.opensuse.org/
    show_url: https://bugzilla.opensuse.org/show_bug.cgi?id=@@@
    regex: "(?:bnc|BNC|bsc|BSC|boo|BOO)\\s*[#:]\\s*(\\d+)"
    user: 
    password: 
    label: boo#@@@
    issues_updated: 2014-12-11 09:48:50.000000000 Z
    enable_fetch: true
method_name: :fetch_issues
args: []

last_error: end of file reached
/usr/lib64/ruby/2.1.0/openssl/buffering.rb:182:in `sysread_nonblock'
/usr/lib64/ruby/2.1.0/openssl/buffering.rb:182:in `read_nonblock'
/usr/lib64/ruby/2.1.0/net/protocol.rb:153:in `rbuf_fill'
/usr/lib64/ruby/2.1.0/net/protocol.rb:134:in `readuntil'
/usr/lib64/ruby/2.1.0/net/protocol.rb:144:in `readline'
/usr/lib64/ruby/2.1.0/net/http/response.rb:39:in `read_status_line'
/usr/lib64/ruby/2.1.0/net/http/response.rb:28:in `read_new'
/usr/lib64/ruby/2.1.0/net/http.rb:1408:in `block in transport_request'
/usr/lib64/ruby/2.1.0/net/http.rb:1405:in `catch'
/usr/lib64/ruby/2.1.0/net/http.rb:1405:in `transport_request'
/usr/lib64/ruby/2.1.0/net/http.rb:1378:in `request'
/usr/lib64/ruby/2.1.0/net/http.rb:1324:in `request_post'
/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:482:in `do_rpc'
/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:286:in `call2'
/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:267:in `call'
/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:606:in `method_missing'
/srv/www/obs/api/app/models/issue_tracker.rb:157:in `fetch_bugzilla_issues'
/srv/www/obs/api/app/models/issue_tracker.rb:204:in `private_fetch_issues'
/srv/www/obs/api/app/models/issue_tracker.rb:140:in `fetch_issues'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/performable_method.rb:26:in `perform'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/backend/base.rb:105:in `block in invoke_job'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `block in initialize'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `execute'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/backend/base.rb:102:in `invoke_job'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:207:in `block (2 levels) in run'
/usr/lib64/ruby/2.1.0/timeout.rb:91:in `block in timeout'
/usr/lib64/ruby/2.1.0/timeout.rb:101:in `call'
/usr/lib64/ruby/2.1.0/timeout.rb:101:in `timeout'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:207:in `block in run'
/usr/lib64/ruby/2.1.0/benchmark.rb:294:in `realtime'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:206:in `run'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:274:in `block in reserve_and_run_one_job'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `block in initialize'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `execute'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:274:in `reserve_and_run_one_job'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:190:in `block in work_off'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:189:in `times'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:189:in `work_off'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:154:in `block (4 levels) in start'
/usr/lib64/ruby/2.1.0/benchmark.rb:294:in `realtime'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:153:in `block (3 levels) in start'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `block in initialize'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `execute'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:152:in `block (2 levels) in start'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:151:in `loop'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:151:in `block in start'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/plugins/clear_locks.rb:7:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:78:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:60:in `block in initialize'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:78:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:78:in `block in add'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:65:in `execute'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/worker.rb:150:in `start'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/command.rb:104:in `run'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/command.rb:92:in `block in run_process'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/controller.rb:70:in `run'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/usr/lib64/ruby/gems/2.1.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/command.rb:90:in `run_process'
/usr/lib64/ruby/gems/2.1.0/gems/delayed_job-4.0.1/lib/delayed/command.rb:79:in `daemonize'
script/delayed_job.api.rb:7:in `<main>'
    run_at: 2014-12-11 12:01:00
 locked_at: NULL
 failed_at: NULL
 locked_by: NULL
     queue: issuetracking

What is this? I don't need any bug tracker updates.

hennevogel commented 9 years ago

Duplicate of #832