Errors when running Rails git tests

[schneems]

These tests are failing on main:

  test 'rails perf:library from git' do
    # BUNDLE_GEMFILE="gemfiles/rails_head.gemfile" bundle exec m test/integration/tasks_test.rb:<linenumber>

    skip unless ENV['USING_RAILS_GIT']

    env = { "TEST_COUNT" => 2, "DERAILED_SCRIPT_COUNT" => 2,
            "SHAS_TO_TEST" => "371f9ad3c4edc5eb12d1ffde94015b6ccc86fd4b,03d949bb0ea1abac04c8cdeba1c55f1a066545b5"}
    puts rake "perf:library", { env: env }
  end

  test "rails perf:library with bad script" do
    # BUNDLE_GEMFILE="gemfiles/rails_head.gemfile" bundle exec m test/integration/tasks_test.rb:<linenumber>

    skip unless ENV['USING_RAILS_GIT']

    error = assert_raises {
      env = { "DERAILED_SCRIPT" => "nopenopenop", "TEST_COUNT" => 2, "DERAILED_SCRIPT_COUNT" => 2,
              "SHAS_TO_TEST" => "371f9ad3c4edc5eb12d1ffde94015b6ccc86fd4b,03d949bb0ea1abac04c8cdeba1c55f1a066545b5"}
      puts rake "perf:library", { env: env }
    }

    assert error.message =~ /nopenopenop:( command)? not found/, "Expected #{error.message} to include /nopenopenop: (command)? not found/ but it did not"
  end

With error:

    Note CF setup setting
rake aborted!
Error while running "git checkout '371f9ad3c4edc5eb12d1ffde94015b6ccc86fd4b' 2>&1": fatal: unable to read tree (371f9ad3c4edc5eb12d1ffde94015b6ccc86fd4b)
/Users/rschneeman/Documents/projects/derailed_benchmarks/lib/derailed_benchmarks/git/in_path.rb:56:in `run!'
/Users/rschneeman/Documents/projects/derailed_benchmarks/lib/derailed_benchmarks/git/in_path.rb:38:in `checkout!'

The purpose of these tests is to exercise the A/B commit testing capabilities of derailed benchmarks. I.e. you point it at a git path, give it two SHAs and a command to run, and it will tell you which is faster.

My best guess is that bundler stopped installing the entire git history somehow, so when it tries to checkout to an older SHA it thinks it doesn't exist.

A path forward could be: Making our own git repo inside of the test and using DERAILED_PATH_TO_LIBRARY to that (or some other way to exercise those code paths). Alternatively we could git clone rails into a tempdir or something but it would be better if it was a consistent location for performance reasons.

It's okay to keep them disabled for now, but it also means we now have less code coverage.

[pcai]

thx for catching this, i meant to open a tracking issue but forgot. and the context is helpful for preserving the intent of the test for whoever ends up renovating this

[pcai]

My best guess is that bundler stopped installing the entire git history somehow, so when it tries to checkout to an older SHA it thinks it doesn't exist.

OK there's pretty good circumstantial evidence that this is indeed what happened: https://github.com/rubygems/rubygems/pull/4475

Here are some version tags someone could use to bisect / verify