search

mootools / mootools-core

MooTools Core Repository
https://mootools.net
2.65k stars 505 forks source link

[Question] the order of platforms for cross-browser testing #2778

Closed junghyun87 closed 7 years ago

junghyun87 commented 7 years ago

Hi, I ask for a research purpose. Has mootools had a criterion to set the order of browsers for cross-browser testing?

The current order of browsers:

    - BUILD='compat'     BROWSER='phantomjs'
    - BUILD='compat'     BROWSER='chrome'
    - BUILD='compat'     BROWSER='firefox'
    - BUILD='compat'     BROWSER='edge'
    - BUILD='compat'     BROWSER='opera12'
    - BUILD='compat'     BROWSER='safari9'
    - BUILD='compat'     BROWSER='safari8'
    - BUILD='compat'     BROWSER='safari7'
    - BUILD='compat'     BROWSER='safari6'
    - BUILD='compat'     BROWSER='ie11'
    - BUILD='compat'     BROWSER='ie10'
    - BUILD='compat'     BROWSER='ie9'
    - BUILD='compat'     BROWSER='ie8'
    - BUILD='compat'     BROWSER='ie7'

    - BUILD='nocompat'   BROWSER='phantomjs'
    - BUILD='nocompat'   BROWSER='chrome'
    - BUILD='nocompat'   BROWSER='firefox'
    - BUILD='nocompat'   BROWSER='edge'
    - BUILD='nocompat'   BROWSER='opera12'
    - BUILD='nocompat'   BROWSER='safari9'
    - BUILD='nocompat'   BROWSER='safari8'
    - BUILD='nocompat'   BROWSER='safari7'
    - BUILD='nocompat'   BROWSER='safari6'
    - BUILD='nocompat'   BROWSER='ie11'
    - BUILD='nocompat'   BROWSER='ie10'
    - BUILD='nocompat'   BROWSER='ie9'
    - BUILD='nocompat'   BROWSER='ie8'
    - BUILD='nocompat'   BROWSER='ie7'

Thanks.

timwienk commented 7 years ago

PhantomJS is at the top because it doesn't require an actual browser, which means for those tests it doesn't need to connect to Sauce Labs and it's faster to execute.

The order of the actual browsers is mostly just "taste" I guess, in the order from the one we hate the least to the one we hate the most, and versions from new to old.

However, you could argue that it's in the order of least expected errors: if something errors in Chrome, Firefox or Edge, there's probably something seriously wrong and we'd see it quicker in this order, whereas the other browsers - especially old IE - sometimes fail for no reason (or even because launching or connecting to the browser fails), so we'd re-run those tests before agreeing that there's an actual error.

junghyun87 commented 7 years ago

@timwienk Thanks for your kind explanation.