Performance tuning audited?

[brightball]

I'm using audited on a couple of tables with a lot of changes, which is creating a slow-growing performance issue as the audit tables grow.

I'm wondering about potential performance tuning options, specifically:

1. Is it possible to off-load all of the audit table processes to background jobs with Resque/Sidekiq?
2. Is it possible to use Mongo exclusively for recording audits to leverage the write-speed perks while leaving the rest of the application on Postgres/MySQL/etc?
3. Is there a way to create multiple audit tables for different models that are being audited to reduce the index sizes?

[mikeatlas]

+1 - I'm interested in this as well. I noticed auditing really slowed down my bulk seed data import time, so I decided to run seed data import with auditing turned off like this:

class Auditing
  def self.disable_auditing
    Auditing.set_auditing(false)
  end

  def self.enable_auditing
    Auditing.set_auditing(true)
  end

  def self.set_auditing(do_auditing)
    Rails.application.eager_load!
    ActiveRecord::Base.descendants.each do |model|
      model.class_attribute :auditing_enabled
      model.auditing_enabled = do_auditing
    end
  end
end

@aramisbear: are you interested in working on a fork with me for this? Generally my app's create/update/delete calls are not so frequent, so perhaps it's not really so critical to focus on after all...

[Tonkpils]

@danielmorrison you seem to be the last one making commits and merging PRs so I'm pulling you in here. If there's anyone else please triage them in.

We are very interested in doing this specifically option 1. Would you be open to accepting a PR that allows for pushing off the auditing records to a background processor?

[danielmorrison]

I'm intrigued by all, but here are some comments:

1. Is it possible to off-load all of the audit table processes to background jobs with Resque/Sidekiq?

I'm open to this if it is opt-in. I don't like the idea of extra dependencies/complexity for people who don't need them.

1. Is it possible to use Mongo exclusively…

I've actually been ripping mongomapper support out, but I'm open to hearing suggestions here. I'm personally more interesting in switching audited to using PostgreSQL's jsonb column type and looking at also storing json for MySQL, etc. but we need an (optional) upgrade path before we can make that happen (We've used this internally via the 1 line change in this branch)..

1. Is there a way to create multiple audit tables for different models that are being audited to reduce the index sizes?

This sounds very interesting. I'd love to see investigation done here to make it easier for people to do this. \cc @laserlemon (who's been asking for this feature too)

[Tonkpils]

Great, the first option is really what I'm looking for and I agree on the opt-in path for it. @jimm any thoughts on possibly creating a PR for this?

[jimm]

Love to. I'll need some time to make it opt-in and create tests. Our current tests are specific to our app's models. Look for a PR soon.

[mikeatlas]

Can't comment anymore personally, I'm no longer working the product which used this gem.

[jimm]

I'm working on the PR. Two questions about how you'd like things handled:

1) Configuration How would you prefer configuration to work? I want to let the user select the async adapter to use globally. I'm thinking it would be nice to configure the class used for queueing in config/initializers/audited.rb. It might look something like

if Rails.env.test?
  Audited.async_class = Audited::Async::Synchronous
else
  Audited.async_class = Audited::Async::Resque
end

(I'm adding instructions in the README for how to develop more async adapters. It's pretty simple.)

2) Class loading How would you like lazy loading to work? Given the above configuration approach, here's how I think it will work: The app developer adds their async gem of choice (for example, resque) to their Gemfile. When Audited::Async::Resque is referenced in the initializer that will load that code, which will in turn load and reference Resque. No require "rescue" should be needed in the audited code. I think that would be sufficient.

[jimm]

Please see PR #288 for a proposed implementation of asynchronous audit creation.

[leonardofalk]

Kinda late now, but I've developed a opt-in solution for this here, I'd appreciate some feedback.