Fixture loading makes before(:all) unpredictable

[virolea]

What Ruby, Rails and RSpec versions are you using?

Ruby version: 3.3.0 Rails version: 8.0 RSpec version: 3.13

Observed behaviour

I am using both factories (with FactoryBot) and fixtures in my application. For some models, I have both factories and fixtures.

To speed some test groups, I am starting to implement some before(:all) hooks in my specs to create test data only once. However, some specs started to fail after moving to this new setup. This only happens when I am running a particular spec, not when running the whole suite.

Here's a simple example that features the problem (a corresponding users.yml fixture file should exist)

require 'rails_helper'

RSpec.describe User, type: :model do
  before(:all) do
    @user = FactoryBot.create(:user)
  end

  after(:all) do
    @user.destroy
  end

  # fails when run separately
  it "exists" do
    expect(User.exists?(@user.id)).to eq true
  end
end

Expected behaviour

The specs should not fail

Can you provide an example reproduction?

From my research, it looks like the fixtures loading happens after the before(:all) setup. Since fixture loading destroys previously set data for a given object, if an instance was created previous to the fixtures loading, it will be wiped out.

Here's an example repo that features the problem https://github.com/virolea/test-app-rspec

bundle install 
bundle exec rspec

Ideally, fixture loading should happen prior to user-defined before(:all) hooks

[pirj]

This is as expected, no? I certainly understand your desire to have clean fixtures on each example, and some (potentially heavy) factories created just once, but this is not trivial to achieve not to say is available out of the box. Have a look at what test-prof has to offer, but I can’t tell off the top of my head if its before_all/let_it_be would work with fixtures.

[virolea]

Hey @pirj thanks for getting back to me so quickly.

This is as expected, no?

From my point of view it was not. I would assume the test setup would be done (and hence fixtures cleaned then loaded) before any test starts. Having my data set in the before(:all) hook wiped out before the test was not something I was expecting and lead to quite a bit of research.

I am not familiar with the code so i don't know the implications behind this design. This is not a blocker as the tests pass at the suite level. Not ideal though.

[pirj]

Can you please provide a minimal repro app/script? Are transactions turned on? Is fixture cleanup done by rolling back the test transaction? Do factories run before the transaction start in before(:all)? If not, why? Is databaseeaner used?

[virolea]

You can use the same one linked in the description: https://github.com/virolea/test-app-rspec

The config of interest here:

# spec/rails_helper.rb
RSpec.configure do |config|
  # [ ...]

  config.global_fixtures = :all
  config.use_transactional_fixtures = true

  # [ ...]
end

Are transactions turned on?

✅ yes

Is fixture cleanup done by rolling back the test transaction?

✅ yes, through use_transactional_fixtures

Do factories run before the transaction start in before(:all)? If not, why?

I am not sure I understand this one. But I don't think this is a factory-related problem as I get the same behavior with plain-old ActiveRecord:

require 'rails_helper'

RSpec.describe User, type: :model do
  before(:all) do
    @user = User.create!(name: "Vincent")
  end

  after(:all) do
    @user.destroy
  end

  # 🔴 Fails 
  it "exists" do
    expect(User.exists?(@user.id)).to eq true
  end
end

Is databaseeaner used?

🚫 no

[pirj]

Sorry for the confusion. Let me rephrase.

What runs earlier, the code that is inside ‘before(:all)’s block, or the very first transaction?

You may get some insights in log/test.log, as usually SQL statements are logged.

[virolea]

The code in the before(:all) block runs first.

However it runs prior to Fixture loading as well, as shown in the logs below:

[pirj]

Can you pinpoint this deletion to some Rails code?

[virolea]

The deletion comes from Fixtures loading. From the docs

The testing environment will automatically load all the fixtures into the database before each test. To ensure consistent data, the environment deletes the fixtures before running the load.

RSpec Rails loads fixtures in

https://github.com/rspec/rspec-rails/blob/609bb83a76c11c67076ff2619fd5f47edd6325ce/lib/rspec/rails/fixture_support.rb#L10

TestFixtures setups fixtures in a before_setup hook that is called via RSpec::Rails::MinitestLifecycleAdapter

https://github.com/rspec/rspec-rails/blob/609bb83a76c11c67076ff2619fd5f47edd6325ce/lib/rspec/rails/adapters.rb#L66-L78

This is the code from Rails itself that setups fixtures:

module ActiveRecord
  module TestFixtures
    extend ActiveSupport::Concern

    def before_setup # :nodoc:
      setup_fixtures
      super
    end

and if you follow deep enough the setup_fixtures call, you end up to the code that deletes tables that have fixtures counterpart.