Prevent caching of result by making pre-check url and runner url unique

[matthijsgroen]

Somehow after a while tests start to return some sort of cached results.

To battle this sort of caching (because it causes mistrust of tests) I shuffled around the reset code. I also made the kill_konacha reset and remove the Capybara session. This helps in that you can now use reload konacha if you would run into this cache issue.

I also changed the session.visit and runner.run urls that they are not the same anymore.

[matthijsgroen]

Please do not merge this pull request yet. It still shows cached results, the only solution is to kill the capybara session after each run. It is a bit slower, but still better than running the rake tasks

[alexgb]

Strange, I haven't noticed this happening. How consistent is this for you? Is it possible to produce a failing test for this?

[matthijsgroen]

I use the plugin the whole day at work. On my Ubuntu machine at home I never had this issue, on the Mac I use at work (a colleague has the same issue on Xubuntu) after a while tests pass or fail, but are somehow just rerunning the old code. If we kill guard and rerun the tests fail or pass as expected again. Restarting konacha:serve in a seperate terminal doesn't even help in this case, so it seems to be some cache that builds up in the capybara-webkit session. It could be that this issue doesn't occur when using a different capybara driver.

For us at the moment only way to ensure correct results is to restart the capybara-webkit session. It is too bad that starting it again takes a performance impact. A different approach could be to restart it directly after the test, so that is is prepared for the next run.

The thing only happens in time so it is hard to produce a test for this. Sometimes it happens after 10 minutes, sometimes after a few hours.

[matthijsgroen]

I'm still trying to improve this flow. I will soon push a change that rebuilds the session after the test run. This will make the next run instant again (however, restarting the capybara session is not taking that long)

[matthijsgroen]

I'm trying to run the tests through selenium now. It will spawn a new firefox window for each test. I will check if I can prevent that behaviour (and also fix a ::Capybara::NotSupportedByDriverError for the 404 sanity check

So consider this as a work in progress :-)

[alexgb]

Have you tried producing this outside of Guard, Konacha? Might be worth seeing if it can be replicated by visiting a URL multiple times in a long running Capybara session.

[matthijsgroen]

I have no idea how I would be able to replicate that behaviour in a non-time-consuming manner. I will just experiment with this branch a bit more. I found some other issues with the plugin as well, status_code is not supported on selenium for instance, I will fix that in a seperate pull-request

[alexgb]

You might try running this https://gist.github.com/alexgb/4945732 in a terminal throughout the day and see if it triggers. Make sure to use the same version of capybara and capybara-webkit that are causing you problems.

[alexgb]

Perhaps this is a bit naive, but maybe could be used as a basis for running a test case against Konacha.

[matthijsgroen]

I ran the gist on both my Mac and Ubuntu machine. Both just kept showing dots.

[matthijsgroen]

I think I will try a different strategy In this matter, and experiment for a different approach. I will close this pull request for now, since this version, although it solves my problems in my daily work, is not clean enough for general release.