The before_start callback fires before running migrations

[Manfred]

In a specific edge case in the PerfCheck CI test suite we built an app that has a blank database when we run PerfCheck.

The following happens:

1. Fetch and hard reset to experiment branch
2. Bundle install
3. Trigger before start callbacks
4. Run migrations
5. (Re)start Rails server etc…

When the before start callbacks attempt to access something introduced by the migrations it will not be able to do so.

One solution would be to trigger the start callbacks after the migrations, but that might introduce other issues.

Potential solutions:

1. Move the callbacks and hope for the best.
2. More fine-grained callbacks (eg. before_all, before_setup, before_each_test_case, etc).
3. Insane callbacks by using event handlers instead of callbacks*

Callbacks

We can have a system where every step in a run can have its own before, around, and after callbacks.

# Configure an around callback so you can decide not to yield for
# a certain step in the process to keep it from happening.
perf_check.steps.around(:start) do |step|
  if perf_check.options.experiment_branch == 'master'
    perf_check.logger.info(
      "Sorry, running benchmarks against master is " \
      "not allowed."
    )
  else
    step.call
  end
end
# Run something afterwards.
perf_check.steps.after(:migrations) do
  puts "Migrations done!"
end
perf_check.steps.around(:request) do |step|
  puts step.call.body
end

class Git
  def checkout(branch)
    perf_check.steps.call(:checkout) do
      git.checkout(branch)
    end
  end
end