Generalizing to the current directory to support all libraries, not just rails

[ericc572]

Currently, this only supports libraries if the user inputs the env variable. Now, we can take the current directory and run the performance tests comparing the two previous commits there.

CC: @schneems

[schneems]

There's a failure

rake aborted!

Don't know how to build task 'library' (See the list of available tasks with `rake --tasks`)

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task_manager.rb:59:in `[]'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:405:in `[]'

/home/travis/build/schneems/derailed_benchmarks/lib/derailed_benchmarks/tasks.rb:7:in `block (2 levels) in <top (required)>'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'

/home/travis/.rvm/rubies/ruby-2.5.5/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:241:in `each'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:241:in `invoke_prerequisites'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'

/home/travis/.rvm/rubies/ruby-2.5.5/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'

/home/travis/.rvm/gems/ruby-2.5.5/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

/home/travis/.rvm/gems/ruby-2.5.5/bin/rake:23:in `load'

/home/travis/.rvm/gems/ruby-2.5.5/bin/rake:23:in `<main>'

/home/travis/.rvm/gems/ruby-2.5.5/bin/ruby_executable_hooks:24:in `eval'

/home/travis/.rvm/gems/ruby-2.5.5/bin/ruby_executable_hooks:24:in `<main>'

Tasks: TOP => perf:library => perf:app

F

Failure:

TasksTest#test_library_branches [/home/travis/build/schneems/derailed_benchmarks/test/integration/tasks_test.rb:24]:

Expected 'env TEST_COUNT=10 DERAILED_SCRIPT_COUNT=2 SHAS_TO_TEST=3054e1d584e7eca110c69a1f8423f2e0866abbf9,80f989aecece1a2b1830e9c953e5887421b52d3c bundle exec rake -f perf.rake perf:library --trace' to return a success status.

Output: 

bin/rails test /home/travis/build/schneems/derailed_benchmarks/test/integration/tasks_test.rb:30

Looks like it's also failing on master https://travis-ci.org/schneems/derailed_benchmarks/builds/623900596?utm_medium=notification&utm_source=github_status

It needs to be fixed, but it's not required before this gets merged in.

We'll need a test that exercises that this task exists and is otherwise syntax error free. You can copy the perf:library test from the integration file and change to perf:app.

We also need some docs in the README.

[schneems]

Awesome! I think we're close.

• Get a changelog in this PR, and add a link to this PR in the changelog entry (which it looks like you just did, great)
• Rebase this against master since I fixed the tests on master

git pull --rebase https://github.com/schneems/derailed_benchmarks master

That should just work ™️ unless there are some merge conflicts.

• Finally, most open-source PRs tend to prefer each PR to be one commit if you contribute to Rails for example. To get this you can run:

git rebase -i HEAD~13

And put a "s" in front of each of your commits but the first one and this will "squash" your commits. Once you're happy, you can force push to your branch.

[ericc572]

Squashed commits and repushed. Odd that travis isn't showing up anymore though

[ericc572]

Not sure why it's failing here: https://travis-ci.org/schneems/derailed_benchmarks/jobs/626860575?utm_medium=notification&utm_source=github_status only for the builds where it's using the rails git gemfile. I see that if the ENV[USING_RAILS_GIT] config var is NOT set, it will skip.

In this test build, however, it will NOT skip since it is being set from that gemfile. Tried to handle this the same way as library does.

[schneems]

This should fix it:

$ git diff HEAD^
diff --git a/test/integration/tasks_test.rb b/test/integration/tasks_test.rb
index 30a5a00..46c8617 100644
--- a/test/integration/tasks_test.rb
+++ b/test/integration/tasks_test.rb
@@ -13,6 +13,13 @@ class TasksTest < ActiveSupport::TestCase
     FileUtils.remove_entry_secure(rails_app_path('tmp'))
   end

+  def run!(cmd)
+    puts "Running: #{cmd}"
+    out = `#{cmd}`
+    raise "Could not run #{cmd}, output: #{out}" unless $?.success?
+    out
+  end
+
   def rake(cmd, options = {})
     assert_success = options.key?(:assert_success) ? options[:assert_success] : true
     env             = options[:env]           || {}
@@ -57,8 +64,9 @@ class TasksTest < ActiveSupport::TestCase

   test 'app' do
     skip unless ENV['USING_RAILS_GIT']
+    run!("cd #{rails_app_path} && git init . && git add . && git commit -m first && git commit --allow-empty -m second")

-    env = { "TEST_COUNT" => 10, "DERAILED_SCRIPT_COUNT" => 2, "SHAS_TO_TEST" => "3054e1d584e7eca110c69a1f8423f2e0866abbf9,80f989aecece1a2b1830e9c953e5887421b52d3c"}
+    env = { "TEST_COUNT" => 10, "DERAILED_SCRIPT_COUNT" => 2 }
     puts rake "perf:app", { env: env }
   end

The fixture app that is being used to run tests on is inside of a top-level git dir, so it does not have it's own git repo. This commit fakes this by creating a new git repo and two empty commits to run against.

[ericc572]

Great! I pushed up and it now passes the test :D