Feature Discussion: gl-matrix 2.0

[toji]

So I've got a bit of free time on my hands and wanted to direct it at something I've been meaning to do for a while: Start the second iteration of the glMatrix library. This issue is to collect feedback on the proposed changes before we go crazy implementing them.

To be clear, the bump to a 2.0 library would indicate breaking of backwards compatibility, and thus several ill considered design decisions/flaws can be corrected without concern about breaking existing usage. The library in it's current form will likely stick around for a while longer and receive a few minor bug fixes, but new features would be 2.0 facing.

A few items that I feel must be part of the refactor:

• The current method of using array-like objects will stick around. The library will continue to create typed arrays by default, but can accept anything numerically indexable.
• As proposed in #41, the semantics for input/output parameters should completely standardized. ALL functions would require an output object where applicable, no more implied change to the first/second param.
• Related to the previous item: Parameter names should be consistent. Names like "vec2" are confusing, since vec2 is also a "type" in gl-matrix. Preferred naming conventions for parameters are "a", "b", and "out", unless a different name would clarify the usage substantially.
• Standardize error handling. There are a couple of cases (matrix inversion, for example) that may have no valid solution. Currently the error handling is "return null and hope they notice." but that's not terribly well defined. Whatever method is used to handle errors, it needs to be well documented and consistent.

In addition there are some things that I would like to see

• I've long lamented the fact that the symantics for matrix multiplication are essentially "backwards" in gl-matrix, as noted by this Google code bug. I feel that any applicable operations should be changed so that the result is a more natural out = a * b instead of the current out = b * a. My only reservation here is that it may make porting code slightly more confusing.
• When gl-matrix was first introduced TypedArrays were fairly new. Now even browsers without WebGL support them, so we should investigate ways to utilize them better. A specific example would be creating a "clone" function that passes the structure to copy into a Float32Array constructor.

Finally, there's a couple of items that have been proposed by the community but that I'm not sold on:

• This Google code bug suggests changing to a non-namespaced pattern, so that mat4.multiply() would be mat4Multiply(). The reason given is that it would be faster (and it would be, if only a little!) but I'm not sure if I like the idea or not.
• I've heard a couple of suggestions that along with requiring an output parameter on all operations we also introduce a set of functions that explicitly alter the first param, most likely named ___Self. a.k.a: vec3.translateSelf(a, [1, 2, 3]); would alter a. This would allow us to retain some current functionality and may even be faster is some cases, but the API is pretty large as-is, and I'm a little reluctant to clutter it further.
• Stop returning the output parameter. This would (theoretically) be faster, but would eliminate the ability to nest functions.

I'm sure there's other items I can add to this list, but that's what I've got off the top of my head. More feedback is appreciated!

[525c1e21-bd67-4735-ac99-b4b0e5262290]

The only thing I wonder about is whether we need a centralized list, or if this issue tracker itself with a 2.0 milestone attached to each decision and maybe a reference to this thread is good enough.

That's up to @toji IMO.

It's his project at heart.

He has to comfortably coordinate this process.

[525c1e21-bd67-4735-ac99-b4b0e5262290]

How about both?

The wishlist is for just that: wishes.

People who don't want to or otherwise can't contribute to the project can collectively express a list of their wishes there.

We can bring the serious and agreeable ones back over to the issue tracker.

This frees up the issue tracker for what hopefully is a majority of pull-requests.

At the end of the day this is up to @toji

[sinisterchipmunk]

At the end of the day this is up to @toji

This. :) But yes, it sounds good to me.

[525c1e21-bd67-4735-ac99-b4b0e5262290]

Basically we all want to help.

We just don't really want to dig through the v2 branch, guessing what needs to be done.

It may seem frivolous to list tasks on such a fine level but I feel it will increase the rate of progress if we can see precisely what needs to be done.

That way effort can go from this discussion (helpful nonetheless) to lines of sweet sweet gl-matrix code.

Until @toji responds it's a nice place to list the conclusions of this discussion in a succinct manner for those of us who just want to bust out some code.

Think of it like a scratchpad.

[toji]

I have no objections to the wishlist, I've often found myself re-scanning this thread and a couple of other places to remind myself what the pending requests/issues are. Thanks @pyrotechnick for putting that together!

I don't have the time at this exact moment to really grok the packaging/build discussion (I'll look at it more over the weekend) but without having read all the arguments I have two high-level comments on the subject:

1) I trust @sinisterchipmunk's judgement in this matter, and am generally happy to go with whatever he feels most comfortable with. 2) After giving it some further though I really do want to have the customized build option, especially since we're talking about things like the vector array functions. That's not an immediate requirement, I think 2.0 can be released without, but ultimately I'd like something along the lines of the jQuery UI download page where you can tick a couple of checkboxes and get only the features you need. Not sure how that works in the context of npm and friends, though. Hm...

And about those vecNArray functions: I'm very well aware of the opportunity for bloat here, and am determined to avoid it. I feel strongly that the Array * Mat functions have real world utility and will be valuable to have optimized, but I really only want to build a few obvious variants, probably just Vec3Array*Mat4 and Vec4Array*Mat4

The forEach function is more interesting in terms of API utility, but I share @sinisterchipmunk's concern about speed. There's some serious benchmarking to be done to determine if it's a worthwhile function at all, and if it can't offer some good "best case" performance then I'm happy to leave it out.

[toji]

FYI: As the result of a couple of minor license issues I've bumped into with glMatrix in the past I've decided that glMatrix v2 will be released under a 2 clause BSD-style license, rather than the ZLIB license glMatrix v1 was released under. This is mostly to ease inclusion into other projects where paranoid project managers apparently don't trust any license but BSD, Apache, or LGPL. (Yes, I've had that conversation before)

For pretty much everyone that uses the library this will mean absolutely nothing changes, so no need to worry. If you for some reason need a different license email me and I'll be glad to accomodate you.

The license text at the top of each file (already checked in) now reads:

/* Copyright (c) 2012, Brandon Jones, Colin MacKenzie IV. All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

  * Redistributions of source code must retain the above copyright notice, this
    list of conditions and the following disclaimer.
  * Redistributions in binary form must reproduce the above copyright notice,
    this list of conditions and the following disclaimer in the documentation 
    and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */

[toji]

Quick status update: Pretty much all of the basic functionality (and the associated unit tests!) are in the 2.0 dev branch now. There's a little more work to do to make sure the build system is doing everything we want (I'd like it to create docs as well), making sure that the release process is what we want, and doing some one-time "Moving from 1.3.7 to 2.0" docs, but all in all I'm thinking that the 2.0 version will be official and "live" by the end of the year! Yay!

Also, an update on an earlier topic: In what will probably be a slightly controversial move I've decided to drop the earlier proposal for vec3Array.transformMat4 but have also decided to keep around the concept of the forEach function. This was after a lot of thought and some JS-voodoo convinced me that it could work out very nicely. The code that will be in the final lib looks like this (with the appropriate changes made for vec2 and vec4 variants)

/**
 * Perform some operation over an array of vec3s.
 *
 * @param {Array} a the array of vectors to iterate over
 * @param {Number} stride Number of elements between the start of each vec3. If 0 assumes tightly packed
 * @param {Number} offset Number of elements to skip at the beginning of the array
 * @param {Number} count Number of vec3s to iterate over. If 0 iterates over entire array
 * @param {Function} fn Function to call for each vector in the array
 * @param {Object} [arg] additional argument to pass to fn
 * @returns {Array} a
 */
vec3.forEach = (function() {
    var vec = new Float32Array(3);

    return function(a, stride, offset, count, fn, arg) {
        var i, l;
        if(!stride) {
            stride = 3;
        }

        if(count) {
            l = Math.min(count * stride, a.length - offset);
        } else {
            l = a.length - offset;
        }

        for(i = offset; i < l; i += stride) {
            vec[0] = a[i]; vec[1] = a[i+1]; vec[2] = a[i+2];
            fn(vec, vec, arg);
            a[i] = vec[0]; a[i+1] = vec[1]; a[i+2] = vec[2];
        }

        return a;
    };
})();

I'm sure that looks a little odd, but it makes a lot more sense when you see it in action, because now you can do this:

var vecs = new Float32Array(300);
// Initialize vecs with data...

vec3.forEach(vecs, 3, 0, 0, vec3.add, [1, 2, 3]);

// Works cleanly with no extra arg
vec3.forEach(vecs, 3, 0, 0, vec3.normalize); 

var m = mat4.create();
mat4.rotateX(m, m, Math.PI);

// And thus we see that we don't need an explicit transform variant any more!
vec3.forEach(vecs, 3, 0, 0, vec3.transformMat4, m);

// And of course you can always do a standard closure
vec3.forEach(vecs, 3, 0, 0, function(out, a) {
    vec3.lerp(out, a, [0, 0, 0], 0.5);
});

So that's actually pretty slick, and I'm pleased with the syntax. Of course, the big question is: How well does it perform? And the answer is.... it's complicated.

According to the jsperf test, in Chrome the forEach version is (surprisingly) significantly faster than even a optimized and special cased loop doing the same operation. I'm guessing that it's just hitting some sweet spot in V8, which is able to optimize the crap out of the functions. Safari turns out to have similar performance characteristics. So yay! Easier to type and faster!

Unfortunately in Firefox the situation is completely flipped. The forEach version is the slowest by a reasonably large margin. Despite that, however, Firefox performance overall was very good, so I'm personally willing to bite that particular bullet and just hope that the awesome team at Mozilla keeps making their JS speedier.

That's all for now!

[SamHowie]

I created a TypeScript definition file for gl-matrix 2.0 for those who are interested.

https://gist.github.com/4268073

I found a couple of small copy & paste bugs in the process (Have already logged them as issues).

The module name may need to be altered pending final namespacing decisions. Also, I have not defined the forEach functions as of yet (Though this should be trivial to do).

Take care.

[toji]

Since glMatrix 2.0 is now live I'm closing this thread up in favor of individual feature requests or bug reports. Thanks to everyone that contributed!