Cached page states — wrong test results

[pepelsbey]

If I'm opening the same URLs (such as http://localhost:7497/tests/?full#5) in multiple tests (via grunt-contrib-connect, grunt-dalek with chrome option, see Gruntfile.js) it seems like Chrome is caching page states or just using the same page for tests with the same URL. I still haven't figure out proper test case, but something is definitely wrong. Here’s why:

1. If all tests above 143 in shortcuts.js are commented, two tests under “Tab” section works fine. Otherwise they fails.
2. The same situation with test on line 36 in navigation.js: it fails if previous tests are not commented
3. Tests could also be “fixed” by moving them at the top of the file.

I should also mention that those three tests also works fine in manual testing (if page is just loaded of course). And .reload() right after .open(…) doesn’t help.

Different tests shouldn’t affect each other results, right?

[pepelsbey]

It seems like PhantomJS is also affected by this “caching” problem: I got the same “all passed” results as in Chrome with tests above commented.

[asciidisco]

That is a really, really weird issue. I will try & set up a minimal testcase to reproduce this. Could take a while...

[pepelsbey]

I’ve fixed one failing test by adding “dumb” test before previous one:

'Right Arrow key is switching to next slide once all Next items becomes Active': function (test) {
    test
        .open('http://0.0.0.0:7497/tests/?full#5')
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE014') // Right
        .assert.attr('[id="6"]', 'class').to.contain('active', 'Next slide is Active')
    .done();
},
'Dumb': function (test) {
    test
        .open('about:blank')
    .done();
},
'Left Arrow key is switching to previous slide when all Next items becomes Active': function (test) {
    test
        .open('http://0.0.0.0:7497/tests/?full#5')
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE014') // Right
        .sendKeys('body', '\uE012') // Left
        .assert.attr('[id="4"]', 'class').to.contain('active', 'Previous slide is Active')
    .done();
},

But it doesn’t work for two more failing tests left.

[pepelsbey]

Have fixed links to tests due to repository refactoring. @asciidisco, is there anything I can do to help you fix this? Test cases, any additional data, etc.

[pepelsbey]

Figured out smaller test case, now with reload()

Run following test:

module.exports = {
    'Test': function (test) {
        test
            .open('http://shwr.me/?full#10')
            .sendKeys('body', '\uE014') // Right
            .reload()
            .assert.doesntExist('[id="10"] .next.active:nth-child(2)', 'Result')
        .done();
    }
};

It will fail, because .active:nth-child(2) is still there after reload but shouldn’t be. Now repeat it manually:

1. Go to http://shwr.me/?full#10
2. Press Right
3. Reload
4. Run it in console: $('[id="10"] .next.active:nth-child(2)')

Console will return null as reload should destroy .active:nth-child(2).

[pepelsbey]

So the problem is related to history: I have History API in my script and use reload, back, forward Dalek’s actions in tests. So they are somehow collide.

[pepelsbey]

Here are some news: I tried to rewrite all my tests using CasperJS and also stuck with reload() and back() problems. But I have fixed all other problems with “cached states” by using clear() destructor at the end of each test, see this.clear() at the end.

casper.test.begin('Non-existing ID in List mode', 2, function suite(test) {
    casper.start('http://0.0.0.0:7497/tests/#404', function() {

        test.assertExists('[id="1"].active', 'Slide #1 is active');
        test.assertUrlMatch(/\/#1/, 'Slide #1 in List mode URL');

    }).run(function() {
        this.clear();
        test.done();
    });
});

So here’s idea: when I run .open(…) I do expect that it would be brand new page loaded just for this particular test. And after test is finished it would be destroyed. But in reality this isn’t happening: both in DalekJS and CasperJS cases. When your page or app requires page states testing it is impossible to do with DalekJS and somehow possible with CasperJS (thanks to clear()).

I think this “brand new page for each test” behavior should be enabled by default. And if you need to run test sequences on one page you can do it inside current test.

[mickaelandrieu]

"So here’s idea: when I run .open(…) I do expect that it would be brand new page loaded just for this particular test. But in reality this isn’t happening" => you're right. In CasperJS, open() does'nt create a new instance of native WebPage object. I think the behavior you expected can be obtained with casper.createPage() or casperjs.clear().

I don't know dalekjs API but I suppose there is possible too.

[izocan]

had the same issue. my workaround was getting everything back in normal state before closing a testsuite. Example: if you have to login in a testcase and dont logout within the same testsuite you cannot rerun your test without errors cuz phantomjs keeps your login data/session or whatever saved. so you have to logout before finishing the test suite or better the test case.

cheers.

[asciidisco]

Interesting issue, I tried to find more about how Webdriver should handle reloads & if there is something in the spec about it, but it turned out that the behaviour isn't described in that detail. I also researched if some other Webdriver libraries had similar issues (at least, if something is listed in the issue tracker, or if they smth. different in the code to handle the state) - but it turned out I wasn't able to find anything...

I would like to ask you for two favours:

1) to try & reproduce the issue using this Webdriver impl. - https://github.com/admc/wd (This will also the driver Dalek will switch to in the future)

2) To get me some link to the code of Caspers clear method, so that I can check what it does internally.

Thanks for your help & research, I really do appreciate that.

[mickaelandrieu]

Hi @asciidisco, actualy an issue is open on CasperJs and we are workin' on it. For now clear method have no impact on native Webpage object of phantomjs.

You can follow discussions through the link shared by @pepelsbey :+1:

[asciidisco]

@mickaelandrieu

Thanks for the answer, but as Casper does not use Webdriver to automate the browser (as Dalek does) there is nothing like Phantoms WebPage object. These are 2 totally different ways of automating browsers. I just wanted to have the link to the exact code that is invoked from Casper, to see waht is going on there on a meta level (are they are clearing the cookies, localstorage or do some other weird foo magic).