Black box test

[odlp]

To prevent regressions, such as #44, it'd be useful to have higher-level tests which invoke bummr in a separate process, with no stubbing.

This first test exercises bummr update. Overview of the steps:

• Creates a new Gemfile with Rake 0.10.0 (outdated)
• Commits and create a new branch
• Runs bummr update
• Checks bummr updated this temporary project to a new version of Rake

[lpender]

Oh one other thing--could we update the README and remove the bit about Aruba doubles ?

Thanks!

[odlp]

Hi Lee,

No worries about the delay. I just rebased off master and noticed this test fails now - Bummr doesn't seem to be committing despite the Gemfile & lockfile updating, and the 'Hello world' Rake command passing.

I'll try and figure out if this a genuine regression or whether the black-box test itself needs updating.

[lpender]

Nice catch! I'm thinking it may be because of the git wrapper that was added... Should we add the command to the CircleCI build so that it fails properly ?

[odlp]

@lpender thanks for figuring out the issue!

Re:

If so, I'm wondering if you thought about, or experimented at all about caching the response i.e. via something like webmock or VCR?

My current thinking is that for it to be a true integration / black-box test that it should hit the network to really prove everything works. The downside is the test suite could become unreliable if the network isn't available (so we should be sparing with these tests). Hopefully the high-level tests provide confidence everything works so it's worth the trade-off vs. potential instability if the network is down.

One tweak we could make is to tag the specs which hit the network:

-describe "bummr update command" do
+describe "bummr update command", :network do
   let(:session) { JetBlack::Session.new(options: { clean_bundler_env: true }) }
   let(:bummr_gem_path) { File.expand_path("../../", __dir__) }

Then say you're offline, developing on a plane, then it's possible to run specs except those which touch the network:

bundle exec rspec --tag '~network'

https://relishapp.com/rspec/rspec-core/docs/command-line/tag-option#exclude-examples-with-a-simple-tag

Would this be useful?