Running tests with caching and tests with stubbed responses is order dependant

[johntrandall]

I've found that for two tests that request from the same domain, where one is cached and the other is stubbed, the stub only works when it is run first.

• This fails with expected to find text "I am not example!" in "Example Domain\nThi....

RSpec.feature "Puffing Billy POC", type: :feature do

  it 'true' do
    expect(true).to be true
  end

  it 'example with no modification', js: true do
    visit 'http://www.example.com/'
    expect(page).to have_content("Example Domain")
  end

  it 'example with stub', js: true do
    proxy.stub('http://www.example.com/').and_return(:text => "I am not example!")
    visit 'http://www.example.com/'
    expect(page).to have_content("I am not example!")
  end
end

• Whereas this passes:

  RSpec.feature "Puffing Billy POC", type: :feature do
  it 'example with stub', js: true do
  proxy.stub('http://www.example.com/').and_return(:text => "I am not example!")
  visit 'http://www.example.com/'
  expect(page).to have_content("I am not example!")
  end
  
  it 'example with no modification', js: true do
  visit 'http://www.example.com/'
  expect(page).to have_content("Example Domain")
  end
  end

• Config as follows:

  
  require 'billy/capybara/rspec'
  Capybara.javascript_driver = :selenium_chrome_billy

Billy.configure do |c| c.cache = true c.persist_cache = true c.cache_path = 'spec/fixtures/puffing_billy/req_cache/' c.certs_path = 'spec/fixtures/puffing_billy/req_certs/' end

[johntrandall]

I've also downgraded back to Chrome 71 and the issue persists.

[johntrandall]

I've also tried adding Billy.proxy.reset_cache before each test without success.

[johntrandall]

tailing the logs and grepping for billy reveals the following when the stub test is first (successful run):

puffing-billy: Proxy listening on http://localhost:49981
puffing-billy: STUB GET for 'http://www.example.com/'
puffing-billy: CACHE KEY for 'http://www.example.com/favicon.ico' is 'get_www.example.com_13ac4b3686b9003f023f633ae532cd200ccd8293'
puffing-billy: CACHE get for 'http://www.example.com/favicon.ico'
puffing-billy: CACHE KEY for 'https://accounts.google.com:443/ListAccounts?gpsia=1&source=durations_metrics&json=standard with body ' '' is 'post_accounts.google.com_3d63af7de140c85cf8ffe2864308a6de3a3d8038_b858cb282617fb0956d960215c8e84d1ccf909c6'
puffing-billy: CACHE KEY for 'https://www.gstatic.com:443/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin' is 'get_www.gstatic.com_9a8ca1000b5c4929140b4f25448698e8c7b04e6c'
puffing-billy: PROXY GET succeeded for 'https://www.gstatic.com:443/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin'
puffing-billy: CACHE KEY for 'http://www.example.com/' is 'get_www.example.com_89e6a0649e06d83370cdf2cbfb05f363934a8d0c'
puffing-billy: CACHE get for 'http://www.example.com/'
puffing-billy: PROXY POST succeeded for 'https://accounts.google.com:443/ListAccounts?gpsia=1&source=durations_metrics&json=standard'

When the stub is second (unsuccessful run), logs are as follows – and the stub entry is missing:

puffing-billy: Proxy listening on http://localhost:50054
puffing-billy: CACHE KEY for 'http://www.example.com/' is 'get_www.example.com_89e6a0649e06d83370cdf2cbfb05f363934a8d0c'
puffing-billy: CACHE get for 'http://www.example.com/'
puffing-billy: CACHE KEY for 'https://accounts.google.com:443/ListAccounts?gpsia=1&source=durations_metrics&json=standard with body ' '' is 'post_accounts.google.com_3d63af7de140c85cf8ffe2864308a6de3a3d8038_b858cb282617fb0956d960215c8e84d1ccf909c6'
puffing-billy: CACHE post for 'https://accounts.google.com:443/ListAccounts?gpsia=1&source=durations_metrics&json=standard'
puffing-billy: CACHE KEY for 'https://www.gstatic.com:443/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin' is 'get_www.gstatic.com_9a8ca1000b5c4929140b4f25448698e8c7b04e6c'
puffing-billy: CACHE get for 'https://www.gstatic.com:443/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin'
puffing-billy: CACHE KEY for 'http://www.example.com/favicon.ico' is 'get_www.example.com_13ac4b3686b9003f023f633ae532cd200ccd8293'
puffing-billy: Received response status code 404 for 'http://www.example.com/favicon.ico'
puffing-billy: PROXY GET succeeded for 'http://www.example.com/favicon.ico'

[ronwsmith]

Hmm, we explicitly allow stubs to override the cache here: https://github.com/oesmith/puffing-billy/blob/master/lib/billy/handlers/request_handler.rb#L20-L21

Instead of reset_cache, maybe try just reset?

[ronwsmith]

@johntrandall is this still an issue for you?

[ronwsmith]

Stale issue, closing.