Chaining timing objects

[tidoust]

Creating an issue to track this idea that Ingar raised on the mailing-list: https://lists.w3.org/Archives/Public/public-webtiming/2015Jul/0006.html

Being able to create a hierarchy of TimingObject objects would make it possible to create timing objects that can process and convert the state vector of another timing object, e.g. to convert the position from one unit to another or to shift or skew the position so that it matches a media timeline.

The spec should define a few of these processing types of TimingObject objects for that to be useful.

I'm not sure how we can create a type of TimingObject that could be associated with custom JavaScript code that does the processing. Without having given it more thoughts, an approach similar to that used in the createAudioWorker method of the Web Audio API [1] where the code runs in a separate Worker could be useful. Incidently, that mechanism is quite similar to the idea of running the timing provider code in a separate JavaScript realm (see issue #10), so there might be a way to merge the two ideas into one.

[1] http://webaudio.github.io/web-audio-api/#widl-AudioContext-createAudioWorker-Promise-AudioWorker

[ingararntzen]

Hi.

I think the ability to chain timing objects to transform motions can be realized using a wrapper object that overrides the relevant part of the timing object API, i.e. query, update and "change" event. This can be done externally, and I don't think it requires any specific support from the timing object. So, I don't think there is any need for any specification here - although the idea could maybe be introduced, for instance in the "programming with timing object" section.

[tidoust]

But then you wouldn't be able to associate that wrapper object directly with a media element, would you?

In other words, the timingsrc property that the spec defines expects a "Platform object" [1], something that JS code cannot produce by itself (other than by calling the TimingObject constructor). A wrapper object, created by authors, would be a "User Object".

What can be done right now is to define a TimingProvider object that wraps a TimingObject, does the transformations, and that is in turn associated with another TimingObject. That is a bit convoluted though, and could in the future become more convoluted if the TimingProvider code ends up being separated (as in #10, although we're not there yet...)

[1] http://heycam.github.io/webidl/#dfn-platform-object

[ingararntzen]

Sorry, forgot about that.

Yes, it would work to generate new provider objects. But it is a bit cumbersome.

So this perhaps indicates the need for a set of predefined TimingWrapperObject (platform objects) ? I have a few in mind

• shiftTimeline - shifts the timeline for the timing object (skew position of source timeline)
• scaleTimeline - stretches the timeline of the timing object (beat/min - frame/sec)
• loopTimeline - make a new timeline that is looped (modulo position of source timeline) -
• rangeTimeline - add say range restrictions to a timing object that does not have range

Though these will probably cover many use cases I doubt that they are exhaustive (what about Timingobject that represents the derivative of the motion of its source TimingObject?)

It would be nice to be able to produce these in JS though.

There is one other approach though...

Lets say that we have two kinds of TimingObjects - one platform object and one user object. Lets call the latter say a MotionObject. Then we have the following chain

Timing Provider -> Motion Object -> Timing Object -> Media Element

The two first are JS and the role of the Timing Object is here to bring external motion from user space into platform space - making it available for all things built-in (media element, possibly sequencer, animation framworks, web audio etc).

However, JS wrappers can now be applied on the Motion Object instead of the Timing Object. The TimingObject is more like a leaf node - and does not support wrapping/chaining.

If you have developed some timing sensitive JS component I suppose you could hook it up to the Motion Object directly and not bother with the Timing Object at all?

What about it? Too much?

[ingararntzen]

Please note that this chaining idea for timing objects is implemented in timingsrc [1,2] under the name timing converters [3].

[1] http://webtiming.github.io/timingsrc/ [2] https://github.com/webtiming/timingsrc [3] http://webtiming.github.io/timingsrc/doc/index.html#timingconverter

[chrisguttandin]

What about adding another overload to the constructor which accepts two functions to modify the vector when querying it or when updating it?

Let's say we want to scale the position by 2 we could then do the following.

const scaledTimingObject = new TimingObject(
    timingObject,
    {
        query: ({ position, ...vector }) => ({ position * 2, ...vector }),
        update: ({ position, ...vector }) => ({ position / 2, ...vector })
    })
);

It would be similar to new timingsrc.ScaleConverter(to, 2) from the demo implementation linked above.

One could also use this to implement similar readymade converters.

const scale = (timingObject, factor) => new TimingObject(
    timingObject,
    {
        query: ({ position, ...vector }) => ({ position * factor, ...vector }),
        update: ({ position, ...vector }) => ({ position / factor, ...vector })
    })
);

It could be used by calling scale(to, 2).

[ingararntzen]

Hi Chris.

I like this. It would provide an effective way of implementing certain types of custom converters, especially skew converter and scale converter. I'm not sure how often one would need other transformations than skew and scale though. However, if you do, it would be pretty neat to be able to solve it like this.

A few comments:

1. This would be a static transformation. The skew converter in timingsrc v3 supports skew to be reset dynamically, and this feature has been quite useful. This is not an argument against the suggestion though, but rather an argument to keep the the existing skew converter around too.
2. If range is defined for the timing object, this must be taken into account. Presumably, the range must be adjusted too, using the provided custom query function. This way, change events resulting from range violations would still be emitted when expected.
3. I think it would be possible to create a query function that somehow messes up some aspect of expected behavior of the timing object. For instance, a query function should probably be required to be well defined within the range and at least piece-wise continuous.

[chrisguttandin]

Yes, you're right. It's probably a bit too powerful.

1. But at least for this use case the usage of arbitrary JavaScript could be an advantage. I think it's possible to dynamically change the value by using a closure. Something like the following should work.

const createSkew = (timingObject, value) => [
    new TimingObject(
        timingObject,
        {
            query: ({ position, ...vector }) => ({ position: position + value, ...vector }),
            update: ({ position, ...vector }) => ({ position: position - value, ...vector })
        })
    ),
    (newValue) => value = newValue
];

const [skew, changeSkew] = createSkew(to, 4);
// skew is a TimingObject with the applied change.
// changeSkew(2) can be used to change the value.

It could probably also be done in an object oriented style.

class SkewedTimingObject extends TimingObject {
    constructor(timingObject, skew) {
        super(
            timingObject,
            {
                query: (vector) => this.query(vector),
                update: (vector) => this.update(vector)
            }
        );

        this.skew = skew;
    }

    query(vector) {
       return { position: position + this.skew, ...vector };
    }

    update(vector) {
       return { position: position - this.skew, ...vector };
    }
}

const skewedTimingObject = new SkewedTimingObject(to, 4);

skewedTimingObject.skew = 2;

2. The algorithm for creating a new TimingObject deals with range violations already. Maybe the same algorithm could be used here too.

   8. If timing's range does not cover the position of the internal vector:
      1. Let timing's internal vector's position be start position or end position, whichever is closest
      2. Let timing's internal vector's velocity and acceleration be 0.0 if the direction of the motion would make the position leave the range immediately.
      3. Set the internal timeout of timing.

3. I'm not sure how that could be enforced. Maybe it's okay to allow people to do weird stuff.