Impractical to cover multiple jQuery versions on developers' systems

[mightyiam]

Our unit test suite doesn't go through all supported jQuery versions.

By default, it tests with jQuery 1.4.4.

It does have a feature where the jQuery version can be requested (an argument in the URI) but practically doing that is virtually impossible because it would take a very, very long time and some tedious address line editing during that time, over and over and over until every supported jQuery version is tested in every supported browser—nope.

Our CI testing implementation does seem to go through jQuery versions (which versions?) but it uses only PhantomJS and that, in turn, uses WebKit, which is very similar in behavior to Blink but it isn't Blink.

I don't see how we can claim to support multiple jQuery versions when we don't test them at all.

For an example of failing test with other-than-default-v1.4.4 jQuery versions try jQuery v1.11.1 in IE8. Or, try jQuery v1.7.1 in IE10.

[winhamwr]

Isn't this a duplicate of #519 ?

A library that only works with a single version of jQuery isn't much use to folks. Such is the nature of dependency management inside a browser. I'm also very unconvinced that supporting multiple versions is much of a burden, but I'm open to being proved wrong. Every time I've encountered an issue specific to a jquery version, it's been an easy fix.

[mightyiam]

Isn't this a duplicate of #519 ?

519 speaks of testing in CI in real browsers, which is not what we currently do.

This issue—which I'm sharpening now—is scoped around the tests that are performed in developer's system, after code edits.

I'm also very unconvinced that supporting multiple versions is much of a burden, but I'm open to being proved wrong.

I'm happy that you're always so open. Otherwise I would have gone nuts. I really appreciate it and see it as a real fine example of modern, constructive, creative environment.

Here is a little explanation of how I think that the testing framework should be designed.

Here's a starting statement: While developers are developing their developments, editing their codes—ya know, they require to know whether their changes are awesome throughout all the browser/jQuery versions matrix.

If we hold the last statement as truth, then we are facing a great difficulty in providing this knowing to the developers.

What currently uncovers whether the developers' changes are awesome throughout this matrix is the unit testing framework that we use QUnit for. And this is done on their own systems.

The reason that we have a great difficulty is because the whole QUnit test suite takes a lot of time, especially in IE7 and IE8. For example, in my overclocked AMD FX-4100 system IE8 takes about 70 seconds. And this is with a single jQuery version.

Until realizing this epiphany of an issue here, I was only testing using the default jQuery v1.4.4 in all browsers. But if the statement is correct, than I must test against the entire browser/jQuery matrix.

Starting with jQuery version 1.4.4 and until the 2.x versions, there are (available in Google's CDN) the following:

1.11.1, 1.11.0, 1.10.2, 1.10.1, 1.10.0, 1.9.1, 1.9.0, 1.8.3, 1.8.2, 1.8.1, 1.8.0, 1.7.2, 1.7.1, 1.7.0, 1.6.4, 1.6.3, 1.6.2, 1.6.1, 1.6.0, 1.5.2, 1.5.1, 1.5.0, 1.4.4

Which sums up as 23 versions.

70 seconds times 23 versions is over 26 minutes. And this is only IE8. IE7 support will be terminated soon. Other browsers are all faster.

In any case, this is unacceptable.

One could argue that the above statement is not correct—developers should only test with the default jQuery version. Ok, let's examine this.

This exposes us to bugs because—as the lords of code and multiple interpreters would have it—jQuery in different versions acts a bit different. And we can't have bugs. I don't want bugs in code that I write. I don't want bugs in users' implementations.

And these are bad bugs because they are hidden. They are like those cockroaches that hide behind stuff and sneak up on you when you're in the shower.

But that's another story.

So, one could argue, in light of this, that, in order to prevent these bugs, we need to catch them. And that same person could argue that in order to prevent them we must detect them. And the same person could argue, sequentially, that, the best way to do that is with an external (CI) testing service, connected to pull requests in GitHub, which will run the QUnit test suite across the real browsers and jQuery versions matrix.

This will be wrong design because then we would have disparity between the test suite that developers run on their systems and the test suite that runs from pull requests in GitHub (CI).

So, for example, if this proposed, "complete matrix", CI test suite, uncovered a bug. And lets say that the bug that it uncovered was, naturally, in a different jQuery version than that which the developer tested with on his own system, than what is the developer to do, then?

Is he to test that jQuery version in his system and use that to fix his code? What if, he does fix it and makes sure that it is still OK for both the default jQuery version and the jQuery version that had the issue and then he proceeds to push that change, and, after some unknown long minutes, discovers that this broke stuff with a different jQuery version?

Add to this the issue of the sheer amount of time that it would take for the CI to generate the result!

In conclusion, it would seem to me that supporting multiple versions of jQuery properly is impractical. We can claim this support but we can't honestly claim it.

Unless some genius idea or change of perception, I'm for supporting a single jQuery version or two, if two will somehow help. More than that might already be too much for practicality.

A library that only works with a single version of jQuery isn't much use to folks. Such is the nature of dependency management inside a browser.

I'm new to all of this so I don't know why you state that.

I guess that minimizing page load is a factor. I can't think of any other factor.

In that regard, I would say that WYMeditor is an editor. And as such, integration of it can be designed such that its assets will download only on pages where it is loaded. On these pages, load time is not as important as the rest of the pages in the host software, probably. How much penalty does an additional jQuery incur on load time?

[winhamwr]

Is he to test that jQuery version in his system and use that to fix his code?

Yup.

after some unknown long minutes, discovers that this broke stuff with a different jQuery version?

That would suck, but isn't particularly likely. jQuery is good about keeping backwards compatibility except in weird edge cases.

As far as the number of minutes, the important thing to realize is that these kind of tests are embarrassingly parallel. We can run the 23 combinations simultaneously, getting results in about the same time as a single run.

it would seem to me that supporting multiple versions of jQuery properly is impractical.

You are greatly overestimating the likelihood of bugs existing exclusively based on jQuery versions and greatly underestimating our ability to create a CI suite that runs quickly in parallel across many machines. Once a project reaches a certain size, it's not uncommon for the test suite to be something that most developers don't fully run on their local machine. They run the portions they think are affected and then rely on the beefy, parallelized CI machines to give them a heads up on other areas to focus.

I'm new to all of this so I don't know why you state that.

Mostly, because making people load a second version of a large library like jQuery on their page is a good way to be labeled as "heavy weight" and "slow." We need to be thinking hard about how to move WYMeditor in the direction of working well on mobile devices. Requiring users to load a second jQuery is not moving that way.

We could make modifications so that we pass in jQuery in a closure to WYMeditor or some such, but I'm not sure of the practical benefit. Once we have a CI suite running our tests against all of the browsers, adding versions of jQuery is easy and we're in the world of using more computer time, which we don't care about. It's cheap.

Maybe we should consider narrowing the list of jQuery versions we support, though. If there's some combination of versions that will cover 80% of our users and we can support a noConflict for the rest, then that's a pretty good compromise. I'm sure jQuery has those kind of usage stats somewhere.

Thoughts?

-Wes

[mightyiam]

after some unknown long minutes, discovers that this broke stuff with a different jQuery version?

That would suck, but isn't particularly likely. jQuery is good about keeping backwards compatibility except in weird edge cases.

It seems that weird edge cases are our game.

So, we'll cook up the real-browser test suite #519 and then we'll return to this with increased wisdom. Cool?

[winhamwr]

cook up the real-browser test suite #519 and then we'll return to this with increased wisdom

+1