search

collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

4.0.3 increases doubles Rails render time, even with Delayed::Worker.delay_jobs = false #703

Closed ajb closed 9 years ago

ajb commented 9 years ago

I'm really sorry to file a bug report that's this vague, but I just spent the entire afternoon tracking down a seemingly impossible performance hit of ~400ms in development to a version bump from 4.0.2 to 4.0.3 of delayed_job. It's definitely be intention to try and track this down more, but I wanted to post immediately since 4.0.3 is only a week old, and others may be running into this.

Here's our Gemfile.lock after pinning the version to 4.0.2:

GIT
  remote: git://github.com/adamjacobbecker/ahoy_email.git
  revision: 30da37b5da111100ce2e8f8725512a639a0be36e
  specs:
    ahoy_email (0.2.1)
      addressable
      nokogiri
      rails

GIT
  remote: git@github.com:dobtco/auth.git
  revision: 7ea28e17ea2c2a8eefcf91d7374472efd6fd04c3
  specs:
    dobt_auth (0.0.1)
      delayed_job (~> 4.0, >= 4.0.0)
      delayed_job_active_record (~> 4.0, >= 4.0.0)
      erector-rails4 (~> 0.1, >= 0.1.0)
      httparty (~> 0.12, >= 0.12.0)
      pg (~> 0.17, >= 0.17.0)
      rails (~> 4.1, >= 4.1.0)
      signed_json (= 0.0.3)
      staccato (~> 0.0, >= 0.0.4)
      storage_unit (~> 0.0, >= 0.0.1)

GEM
  remote: https://rubygems.org/
  remote: https://rails-assets.org/
  specs:
    actionmailer (4.1.4)
      actionpack (= 4.1.4)
      actionview (= 4.1.4)
      mail (~> 2.5.4)
    actionpack (4.1.4)
      actionview (= 4.1.4)
      activesupport (= 4.1.4)
      rack (~> 1.5.2)
      rack-test (~> 0.6.2)
    actionview (4.1.4)
      activesupport (= 4.1.4)
      builder (~> 3.1)
      erubis (~> 2.7.0)
    active_model_serializers (0.8.1)
      activemodel (>= 3.0)
    activemodel (4.1.4)
      activesupport (= 4.1.4)
      builder (~> 3.1)
    activerecord (4.1.4)
      activemodel (= 4.1.4)
      activesupport (= 4.1.4)
      arel (~> 5.0.0)
    activesupport (4.1.4)
      i18n (~> 0.6, >= 0.6.9)
      json (~> 1.7, >= 1.7.7)
      minitest (~> 5.1)
      thread_safe (~> 0.1)
      tzinfo (~> 1.1)
    addressable (2.3.6)
    annotate (2.6.5)
      activerecord (>= 2.3.0)
      rake (>= 0.8.7)
    api-pagination (3.0.2)
    arel (5.0.1.20140414130214)
    better_errors (1.1.0)
      coderay (>= 1.0.0)
      erubis (>= 2.6.6)
    binding_of_caller (0.7.2)
      debug_inspector (>= 0.0.1)
    builder (3.2.2)
    callsite (0.0.11)
    cancan (1.6.10)
    capybara (2.4.1)
      mime-types (>= 1.16)
      nokogiri (>= 1.3.3)
      rack (>= 1.0.0)
      rack-test (>= 0.5.4)
      xpath (~> 2.0)
    capybara-webkit (1.3.0)
      capybara (>= 2.0.2, < 2.5.0)
      json
    carrierwave (0.10.0)
      activemodel (>= 3.2.0)
      activesupport (>= 3.2.0)
      json (>= 1.7)
      mime-types (>= 1.16)
    celluloid (0.15.2)
      timers (~> 1.1.0)
    charlock_holmes (0.7.3)
    chronic (0.10.2)
    codemirror-rails (4.2)
      railties (>= 3.0, < 5)
    coderay (1.1.0)
    coffee-rails (4.0.1)
      coffee-script (>= 2.2.0)
      railties (>= 4.0.0, < 5.0)
    coffee-script (2.2.0)
      coffee-script-source
      execjs
    coffee-script-source (1.7.0)
    colorist (0.0.2)
    cookiejar (0.3.2)
    crass (0.2.0)
    css_parser (1.3.5)
      addressable
    csv_builder (2.1.1)
      actionpack (>= 3.0.0)
    database_cleaner (1.3.0)
    debug_inspector (0.0.2)
    delayed_job (4.0.2)
      activesupport (>= 3.0, < 4.2)
    delayed_job_active_record (4.0.2)
      activerecord (>= 3.0, < 4.2)
      delayed_job (>= 3.0, < 4.1)
    diff-lcs (1.2.5)
    docile (1.1.5)
    dotenv (0.11.1)
      dotenv-deployment (~> 0.0.2)
    dotenv-deployment (0.0.2)
    dotenv-rails (0.11.1)
      dotenv (= 0.11.1)
    eco (1.0.0)
      coffee-script
      eco-source
      execjs
    eco-source (1.1.0.rc.1)
    ejs (1.1.1)
    em-http-request (1.1.2)
      addressable (>= 2.3.4)
      cookiejar
      em-socksify (>= 0.3)
      eventmachine (>= 1.0.3)
      http_parser.rb (>= 0.6.0)
    em-socksify (0.3.0)
      eventmachine (>= 1.0.0.beta.4)
    email_reply_parser (0.5.7)
    erector-rails4 (0.1.3)
      rails (>= 3.0.0)
      treetop
    erubis (2.7.0)
    eventmachine (1.0.3)
    excon (0.37.0)
    execjs (2.2.1)
    factory_girl (4.4.0)
      activesupport (>= 3.0.0)
    factory_girl_rails (4.4.1)
      factory_girl (~> 4.4.0)
      railties (>= 3.0.0)
    ffaker (1.24.0)
    ffi (1.9.3)
    filterer (0.3.3)
      rails (>= 4.0.0)
    fog (1.22.1)
      fog-brightbox
      fog-core (~> 1.22)
      fog-json
      ipaddress (~> 0.5)
      nokogiri (~> 1.5, >= 1.5.11)
    fog-brightbox (0.1.1)
      fog-core (~> 1.22)
      fog-json
      inflecto
    fog-core (1.22.0)
      builder
      excon (~> 0.33)
      formatador (~> 0.2)
      mime-types
      net-scp (~> 1.1)
      net-ssh (>= 2.1.3)
    fog-json (1.0.0)
      multi_json (~> 1.0)
    foreigner (1.6.1)
      activerecord (>= 3.0.0)
    formatador (0.2.5)
    friendly_id (5.0.4)
      activerecord (>= 4.0.0)
    gene_pool (1.4.1)
      thread_safe
    geocoder (1.2.2)
    guard (2.6.1)
      formatador (>= 0.2.4)
      listen (~> 2.7)
      lumberjack (~> 1.0)
      pry (>= 0.9.12)
      thor (>= 0.18.1)
    guard-rspec (4.2.10)
      guard (~> 2.1)
      rspec (>= 2.14, < 4.0)
    guard-teaspoon (0.8.0)
      guard (~> 2.2)
      teaspoon (>= 0.8.0)
    hike (1.2.3)
    honeybadger (1.16.2)
      json
    htmlentities (4.3.2)
    http_parser.rb (0.6.0)
    httparty (0.13.1)
      json (~> 1.8)
      multi_xml (>= 0.5.2)
    httpclient (2.3.4.1)
    i18n (0.6.11)
    immigrant (0.1.8)
      activerecord (>= 3.0)
      foreigner (>= 1.2.1)
    inflecto (0.0.2)
    ipaddress (0.8.0)
    jquery-turbolinks (2.0.2)
      railties (>= 3.1.0)
      turbolinks
    json (1.8.1)
    judge (2.0.4)
      rails (>= 3.1)
    judge-simple_form (0.4.0)
      judge (~> 2.0)
      simple_form (>= 2.0)
    kaminari (0.16.1)
      actionpack (>= 3.0.0)
      activesupport (>= 3.0.0)
    launchy (2.4.2)
      addressable (~> 2.3)
    leaflet-rails (0.7.3)
    letter_opener (1.2.0)
      launchy (~> 2.2)
    liquid (2.6.1)
    listen (2.7.9)
      celluloid (>= 0.15.2)
      rb-fsevent (>= 0.9.3)
      rb-inotify (>= 0.9)
    lumberjack (1.0.9)
    mail (2.5.4)
      mime-types (~> 1.16)
      treetop (~> 1.4.8)
    mandrill-rails (1.0.2)
      activesupport (>= 3.0.3)
    memoist (0.9.3)
    meta_request (0.3.4)
      callsite (~> 0.0, >= 0.0.11)
      rack-contrib (~> 1.1)
      railties (>= 3.0.0, < 5.0.0)
    method_source (0.8.2)
    mime-types (1.25.1)
    mini_magick (3.7.0)
      subexec (~> 0.2.1)
    mini_portile (0.6.0)
    minitest (5.4.1)
    multi_json (1.10.1)
    multi_xml (0.5.5)
    net-scp (1.2.1)
      net-ssh (>= 2.6.5)
    net-ssh (2.9.1)
    nokogiri (1.6.3.1)
      mini_portile (= 0.6.0)
    nokogumbo (1.1.9)
      nokogiri
    non-stupid-digest-assets (1.0.4)
    persistent_http (1.0.6)
      gene_pool (>= 1.3)
    persistent_httparty (0.1.1)
      httparty (~> 0.9)
      persistent_http
    pg (0.17.1)
    pg_search (0.7.4)
      activerecord (>= 3.1)
      activesupport (>= 3.1)
      arel
    polyglot (0.3.5)
    possessive (1.0.1)
    premailer (1.8.2)
      css_parser (>= 1.3.5)
      htmlentities (>= 4.0.0)
    premailer-rails (1.7.0)
      actionmailer (>= 3, < 5)
      premailer (~> 1.7, >= 1.7.9)
    pretty_id (0.0.2)
      rails
    pry (0.10.0)
      coderay (~> 1.1.0)
      method_source (~> 0.8.1)
      slop (~> 3.4)
    pusher (0.12.0)
      httpclient (~> 2.3.0)
      multi_json (~> 1.0)
      signature (~> 0.1.6)
    rack (1.5.2)
    rack-contrib (1.1.0)
      rack (>= 0.9.1)
    rack-cors (0.2.9)
    rack-ssl-enforcer (0.2.7)
    rack-test (0.6.2)
      rack (>= 1.0)
    rails (4.1.4)
      actionmailer (= 4.1.4)
      actionpack (= 4.1.4)
      actionview (= 4.1.4)
      activemodel (= 4.1.4)
      activerecord (= 4.1.4)
      activesupport (= 4.1.4)
      bundler (>= 1.3.0, < 2.0)
      railties (= 4.1.4)
      sprockets-rails (~> 2.0)
    rails-assets-ajb-sanitize (1.0.0)
    rails-assets-backbone (1.1.2)
      rails-assets-underscore (>= 1.5.0)
    rails-assets-backbone-deep-model (0.10.4)
    rails-assets-beforeunload.js (0.0.1)
    rails-assets-font-awesome (3.2.1)
    rails-assets-formrenderer-base (0.0.13)
      rails-assets-ajb-sanitize (= 1.0.0)
      rails-assets-backbone (~> 1.1.2)
      rails-assets-backbone-deep-model (~> 0.10.4)
      rails-assets-beforeunload.js (= 0.0.1)
      rails-assets-font-awesome (= 3.2.1)
      rails-assets-iso-country-names (= 0.0.1)
      rails-assets-jquery (>= 1.5)
      rails-assets-jquery-form (~> 3.46.0)
      rails-assets-leaflet (~> 0.7.3)
      rails-assets-rivets (= 0.5.12)
      rails-assets-store.js (~> 1.3.16)
      rails-assets-underscore (~> 1.6.0)
      rails-assets-underscore.sanitize.js (= 0.0.2)
      rails-assets-underscore.simpleformat.js (= 0.0.2)
      rails-assets-underscore.string (~> 2.3.3)
      rails-assets-underscore.toboolean.js (= 0.0.1)
    rails-assets-iso-country-names (0.0.1)
    rails-assets-jeresig--jquery.hotkeys (0.1.0)
    rails-assets-jquery (2.1.1)
    rails-assets-jquery-form (3.46.0)
      rails-assets-jquery (>= 1.5)
    rails-assets-jquery-ui (1.10.3)
      rails-assets-jquery (>= 1.6)
    rails-assets-jquery-ujs (1.0.1)
      rails-assets-jquery (> 1.8)
    rails-assets-leaflet (0.7.3)
    rails-assets-moment (2.0.0)
    rails-assets-purl (2.3.1)
    rails-assets-rangeslider.js (0.3.3)
      rails-assets-jquery (>= 1.9.0)
    rails-assets-rivets (0.5.12)
    rails-assets-store.js (1.3.16)
    rails-assets-underscore (1.6.0)
    rails-assets-underscore.sanitize.js (0.0.2)
      rails-assets-ajb-sanitize (= 1.0.0)
      rails-assets-underscore (~> 1.6.0)
    rails-assets-underscore.simpleformat.js (0.0.2)
      rails-assets-underscore (~> 1.6.0)
    rails-assets-underscore.string (2.3.3)
    rails-assets-underscore.toboolean.js (0.0.1)
      rails-assets-underscore (~> 1.6.0)
    railties (4.1.4)
      actionpack (= 4.1.4)
      activesupport (= 4.1.4)
      rake (>= 0.8.7)
      thor (>= 0.18.1, < 2.0)
    rake (10.3.2)
    rb-fsevent (0.9.4)
    rb-inotify (0.9.5)
      ffi (>= 0.5.0)
    redis (3.1.0)
    redis-actionpack (4.0.0)
      actionpack (~> 4)
      redis-rack (~> 1.5.0)
      redis-store (~> 1.1.0)
    redis-activesupport (4.0.0)
      activesupport (~> 4)
      redis-store (~> 1.1.0)
    redis-rack (1.5.0)
      rack (~> 1.5)
      redis-store (~> 1.1.0)
    redis-rails (4.0.0)
      redis-actionpack (~> 4)
      redis-activesupport (~> 4)
      redis-store (~> 1.1.0)
    redis-store (1.1.4)
      redis (>= 2.2)
    remote_syslog_logger (1.0.3)
      syslog_protocol
    rinku (1.7.3)
    rspec (3.0.0)
      rspec-core (~> 3.0.0)
      rspec-expectations (~> 3.0.0)
      rspec-mocks (~> 3.0.0)
    rspec-core (3.0.2)
      rspec-support (~> 3.0.0)
    rspec-expectations (3.0.2)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.0.0)
    rspec-its (1.0.1)
      rspec-core (>= 2.99.0.beta1)
      rspec-expectations (>= 2.99.0.beta1)
    rspec-mocks (3.0.2)
      rspec-support (~> 3.0.0)
    rspec-rails (3.0.1)
      actionpack (>= 3.0)
      activesupport (>= 3.0)
      railties (>= 3.0)
      rspec-core (~> 3.0.0)
      rspec-expectations (~> 3.0.0)
      rspec-mocks (~> 3.0.0)
      rspec-support (~> 3.0.0)
    rspec-retry (0.3.0)
      rspec
    rspec-support (3.0.2)
    ruby-ole (1.2.11.7)
    rubyzip (1.1.6)
    sanitize (3.0.0)
      crass (~> 0.2.0)
      nokogiri (>= 1.4.4)
      nokogumbo (= 1.1.9)
    sass (3.2.19)
    sass-rails (4.0.3)
      railties (>= 4.0.0, < 5.0)
      sass (~> 3.2.0)
      sprockets (~> 2.8, <= 2.11.0)
      sprockets-rails (~> 2.0)
    sequenced (1.5.0)
      activerecord (>= 3.0)
      activesupport (>= 3.0)
    signature (0.1.7)
    signed_json (0.0.3)
      json
    simple_form (3.0.2)
      actionpack (~> 4.0)
      activemodel (~> 4.0)
    simple_form-dropdown_select (0.0.3)
      coffee-script
      rspec-rails
      simple_form
      stylus
    simplecov (0.9.0)
      docile (~> 1.1.0)
      multi_json
      simplecov-html (~> 0.8.0)
    simplecov-html (0.8.0)
    slop (3.5.0)
    spreadsheet (0.9.7)
      ruby-ole (>= 1.0)
    spring (1.1.3)
    spring-commands-rspec (1.0.2)
      spring (>= 0.9.1)
    spring-commands-teaspoon (0.0.2)
      spring (>= 0.9.1)
    sprockets (2.11.0)
      hike (~> 1.2)
      multi_json (~> 1.0)
      rack (~> 1.0)
      tilt (~> 1.1, != 1.3.0)
    sprockets-rails (2.1.4)
      actionpack (>= 3.0)
      activesupport (>= 3.0)
      sprockets (~> 2.8)
    sqlite3 (1.3.9)
    staccato (0.1.1)
    storage_unit (0.0.1)
      rails (~> 4.1, >= 4.1.0)
    stylus (1.0.1)
      execjs
      stylus-source
    stylus-source (0.42.2)
    subexec (0.2.3)
    sync (0.3.1)
      em-http-request
    syslog_protocol (0.9.2)
    teaspoon (0.8.0)
      railties (>= 3.2.5, < 5)
    temping (3.2.0)
      activerecord (>= 3.1)
      activesupport (>= 3.1)
      sqlite3 (~> 1.3.7)
    terminal-notifier-guard (1.5.3)
    test_after_commit (0.2.3)
    thor (0.19.1)
    thread_safe (0.3.4)
    tilt (1.4.1)
    timecop (0.7.1)
    timers (1.1.0)
    treetop (1.4.15)
      polyglot
      polyglot (>= 0.3.1)
    truncate_html (0.9.2)
    turbolinks (2.2.2)
      coffee-rails
    tzinfo (1.2.2)
      thread_safe (~> 0.1)
    uglifier (2.5.1)
      execjs (>= 0.3.0)
      json (>= 1.8.0)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.6)
    versionist (1.3.0)
      rails (>= 3)
      yard (~> 0.7)
    whenever (0.9.2)
      activesupport (>= 2.3.4)
      chronic (>= 0.6.3)
    wicked_pdf (0.11.0)
      rails
    wkhtmltopdf-binary (0.9.9.3)
    xpath (2.0.0)
      nokogiri (~> 1.3)
    yard (0.8.7.4)
    zeroclipboard-rails (0.1.0)
      railties (>= 3.1)

PLATFORMS
  ruby

DEPENDENCIES
  active_model_serializers (= 0.8.1)
  ahoy_email!
  annotate
  api-pagination
  better_errors
  binding_of_caller
  cancan
  capybara
  capybara-webkit
  carrierwave
  charlock_holmes
  codemirror-rails
  coffee-rails
  colorist
  cookiejar
  csv_builder
  database_cleaner
  delayed_job (= 4.0.2)
  delayed_job_active_record
  dobt_auth!
  dotenv-rails
  eco
  ejs
  email_reply_parser
  erector-rails4
  execjs
  factory_girl_rails
  ffaker
  filterer
  fog
  foreigner
  friendly_id
  geocoder
  guard-rspec
  guard-teaspoon
  honeybadger
  httparty
  immigrant
  jquery-turbolinks
  judge
  judge-simple_form
  kaminari
  launchy
  leaflet-rails
  letter_opener
  liquid
  mandrill-rails
  memoist
  meta_request
  mime-types
  mini_magick
  non-stupid-digest-assets
  persistent_httparty
  pg
  pg_search
  possessive
  premailer-rails
  pretty_id
  pusher
  rack-cors
  rack-ssl-enforcer
  rails (= 4.1.4)
  rails-assets-ajb-sanitize
  rails-assets-beforeunload.js
  rails-assets-formrenderer-base (= 0.0.13)
  rails-assets-jeresig--jquery.hotkeys
  rails-assets-jquery-form
  rails-assets-jquery-ui (= 1.10.3)
  rails-assets-jquery-ujs
  rails-assets-moment (= 2.0.0)
  rails-assets-purl
  rails-assets-rangeslider.js
  rails-assets-underscore
  redis-rails
  remote_syslog_logger
  rinku
  rspec-its
  rspec-rails
  rspec-retry
  rubyzip (>= 1.0.0)
  sanitize
  sass-rails
  sequenced (= 1.5.0)
  simple_form
  simple_form-dropdown_select
  simplecov
  spreadsheet
  spring
  spring-commands-rspec
  spring-commands-teaspoon
  storage_unit
  stylus
  sync
  teaspoon
  temping
  terminal-notifier-guard
  test_after_commit
  timecop
  truncate_html
  turbolinks
  uglifier
  unf
  versionist
  whenever
  wicked_pdf
  wkhtmltopdf-binary
  zeroclipboard-rails
albus522 commented 9 years ago

We removed a bunch of hacks to YAML that were no longer needed by DJ. These might have been helping your application.

ajb commented 9 years ago

Great to know, thanks for the tip. Will let you know what we find. On Sep 12, 2014 3:51 PM, "David Genord II" notifications@github.com wrote:

We removed a bunch of hacks to YAML that were no longer needed by DJ. These might have been helping your application.

— Reply to this email directly or view it on GitHub https://github.com/collectiveidea/delayed_job/issues/703#issuecomment-55452393 .

ajb commented 9 years ago

Hey @albus522, thanks again for the pointer. I've indeed tracked the source to psych_ext.rb. Can you tell me any more about the hacks that were removed?

Here's a quick process time profiling of a basic HTTP request, on 4.0.2 and 4.0.3.

albus522 commented 9 years ago

https://github.com/collectiveidea/delayed_job/compare/v4.0.2...v4.0.3#files_bucket will give you the changes between the 2 versions

ajb commented 9 years ago

Sorry, I've got that, I'm just wondering about this history of the Psych hacks. I've looked at the blame and file history, but I'm struggling to find the backstory.

albus522 commented 9 years ago

All hacks were in place to fix issues with old versions of ruby working with YAML. The hacks that are still there are issues that still exist with current versions of ruby. You can remove the contents of psych_ext and watch things blow up.

albus522 commented 9 years ago

None of our hacks were performance related. They were all functionality fixes.

chrisb87 commented 9 years ago

We've also seen a dramatic performance hit for our app when upgrading to 4.0.3. Not just http requests, if I open a rails console and reference one of the models the console hangs for a good 30 seconds before responding. After that it's smooth sailing.

albus522 commented 9 years ago

As of 4.0.4 YAML is completely native except when we are loading the job payload. Any performance discussions should be had with the psych gem development team.

ajb commented 9 years ago

Completely understandable. @chrisb87, have you gotten anywhere with understanding where your performance hit is coming from? I'd love to team up and tackle this.

chrisb87 commented 9 years ago

@ajb, I haven't, for now I've just locked my projects to dj 4.0.2