cancelScheduledValues should set the current value as a parameter change

[cwilso]

cancelScheduledValues should explicitly create a parameter value change in the event list of the current value, at that time. This will enable smooth ramps from the current value.

[karlt]

This would change the behavior a bit if there is a setTargetAtTime event in effect at the startTime for cancelScheduleValues. That's probably OK because another setTargetAtTime event can be scheduled at startTime, if the intention is to complete the exponential approach to the existing target.

I wonder what would be best if startTime for cancelScheduleValues is within the interval of an existing setValueCurveAtTime event. I guess it would be most useful if cancelScheduleValues succeeds, and the current value for the parameter value change were taken from the value curve.

I'm not sure we need the additional complexity for the sake of truncating the value curve, but the NOT_SUPPORTED_ERR requirements around value curve durations overlapping other events are a bit awkward due to floating point precision and the start + duration specification of the end of the interval. It may not be possible to schedule a subsequent event exactly at the end of the value curve interval.

Would the current value for the cancelScheduledValues event be based on the events currently scheduled? Or would events added later but with times before the cancelScheduledValues event change the value for the cancelScheduledValues event? If the latter, then I assume events scheduled before but with times after the cancelScheduledValues event are not remembered to calculate the new current value?

I wonder whether it would be helpful to specify that if the startTime for cancelScheduledValues is before currentTime, then the new event is scheduled at currentTime. This is similar to the situation with AudioBufferSourceNode start times. Do we want to save people from jumps in the computed value (which would be induced by changing the events in the past), or do we want to provide for values in the future as specified (which would not be quite the case if moving the event time)?

[joeberkovitz]

TPAC resolution: We will define cancelScheduledValues as deleting any extant automation events scheduled at/after the automation time, Clarify the spec to point out that this may cause discontinuities due to the disappearance of ramps whose endTime is scheduled after the cancellation point.

When cancelling automation and there is a desire to continue the parameter from the current value, recommend the following sequence:

var currentValue = param.value;
param.cancelScheduledValues(ctx.currentTime);
setValueAtTime(currentValue, ctx.currentTime);

[mark-buer]

That is a very disappointing decision.

Perhaps the spec should also state this advisory to consumers of the API: "The automation methods are designed for set-and-forget automations. If your use-case demands glitch-free audio in the face of changing automations, such as those required in many interactive applications, the automation methods are probably not useful." Or words to that effect.

Out of curiosity, what happens if an audio render event occurs part-way-through the suggested value continuation code sequence?

var currentValue = param.value;
// !!! Audio render occurs *right now* on the audio thread !!!
param.cancelScheduledValues(ctx.currentTime);
setValueAtTime(currentValue, ctx.currentTime);

My use-case for cancel-and-hold: Assume that long-running gain automations (of the order of seconds) are in effect on many (lets say seven or eight) gain nodes of a moderately complex audio graph. Each gain node requires a separate automation curve. The number of gain nodes changes with respect to time (over the order of tens of seconds). Unpredictable user interactions trigger a change of all existing automations. It is necessary to change the automations in a way that avoids clicks and pops. It is acceptable for there to be a slight lag (of the order of tens of milliseconds) between user interaction and the change of automation.

Solving that use-case is trivial if cancel-and-hold exists, but extremely complex if it doesn't exist. Right now I'm dreaming up ways of cross fading between old gain nodes with old automations and new gain nodes with new automations... or perhaps a super-wide audio worker (with tens of channels) fed into a channel splitter that will provide the automation values... or... dunno. Not easy. Very complex.

A lack of cancel-and-hold adversely affects the utility of the automation functions, especially for interactive applications where those automations need to change without audible glitches. I'm sure it will complicate the implementation and the spec, but without cancel-and-hold, that complexity falls squarely on the shoulders of the consumers of the API. Please reconsider.

[mrkishi]

I agree with @mark-buer.

If the intention was to keep cancelScheduledValues simple, what about a separate method for the "hold" use-case? Something like AudioParam.splitAt(time) could insert a new event to the automation timeline, taking into account the actual value at that time.

This would allow us to smoothly modify the automation curve between time and next event, or simply cancelScheduledValues(t) where time < t < next event.

[rtoy]

Although not captured in Joe's message, we did try hard to do some kind of cancel and hold, but the issue boiled down to how to specify that and also when and how to remove future events while still allowing future events to be added after the call to cancelScheduledEvents. And this is all complicated by how to handle a cancelScheduledEvents when the specified time occurs in the middle of an automation that is already in progress.

On Mon, Oct 26, 2015 at 3:36 PM, Mark Buer notifications@github.com wrote:

That is a very disappointing decision.

Perhaps the spec should also state this advisory to consumers of the API: "The automation methods are designed for set-and-forget automations. If your use-case demands glitch-free audio in the face of changing automations, such as those required in many interactive applications, the automation methods are probably not useful." Or words to that effect.

Out of curiosity, what happens if an audio render event occurs part-way-through the suggested value continuation code sequence?

var currentValue = param.value;// !!! Audio render occurs right now on the audio thread !!! param.cancelScheduledValues(ctx.currentTime); setValueAtTime(currentValue, ctx.currentTime);

My use-case for cancel-and-hold: Assume that long-running gain automations (of the order of seconds) are in effect on many (lets say seven or eight) gain nodes of a moderately complex audio graph. Each gain node requires a separate automation curve. The number of gain nodes changes with respect to time (over the order of tens of seconds). Unpredictable user interactions trigger a change of all existing automations. It is necessary to change the automations in a way that avoids clicks and pops. It is acceptable for there to be a slight lag (of the order of tens of milliseconds) between user interaction and the change of automation.

Solving that use-case is trivial if cancel-and-hold exists, but extremely complex if it doesn't exist. Right now I'm dreaming up ways of cross fading between old gain nodes with old automations and new gain nodes with new automations... or perhaps a super-wide audio worker (with tens of channels) fed into a channel splitter that will provide the automation values... or... dunno. Not easy. Very complex.

A lack of cancel-and-hold adversely affects the utility of the automation functions, especially for interactive applications where those automations need to change without audible glitches. I'm sure it will complicate the implementation and the spec, but without cancel-and-hold, that complexity falls squarely on the shoulders of the consumers of the API. Please reconsider.

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/344#issuecomment-151304617 .

Ray

[cwilso]

Let me ask a more targeted question - would a non-SCHEDULED "cancel and hold" method fulfill your scenario? That is, a single method that clears all future automation immediately (aka at the beginning of the next block), and sets a schedule point of the current time and current value?

void cancelAutomation( void );

[mark-buer]

Yes, a non-SCHEDULED "cancel and hold" method would fulfil the cancellation requirements of my scenario.

Would it be possible for API consumers to invoke cancelAutomation, followed by immediate invocations to add new automations (scheduled for some time > currentTime)? For example, would this:

var currentTime = audioContext.currentTime;
param.cancelAutomation();
param.linearRampToValueAtTime(v1, currentTime + t1); // where t1 is "suitably" large
param.linearRampToValueAtTime(v2, currentTime + t2); // where t2 > t1
// ... etc

just do the right thing? Or, would the API consumer need to perform additional synchronisation steps so as to avoid adding the new automations prior to the cancellation actually being performed by the audio thread?

For my use-case (see code sample above), the overall accuracy and timing of the resulting envelope is not as important as a click-and-pop free audio stream. As a result, the slope of the post-cancellation linear automation curve is not critical. I can choose t1 and v1 to give a suitably flat slope, with allowances made for the unpredictable time (and param value) of cancellation.

So for my particular application: the lack of control of the precise time of cancellation is okay, the lack of predictability of the time of cancellation is also okay. and the lack of knowledge of the value at the time of cancellation is okay.

I wonder how many other applications would find this API adequate, versus how many would find it inadequate?

[cwilso]

Yes, you'd be able to immediately call other automation functions.

On Wed, Oct 28, 2015 at 6:26 AM, Mark Buer notifications@github.com wrote:

Yes, a non-SCHEDULED "cancel and hold" method would fulfil the cancellation requirements of my scenario.

Would it be possible for API consumers to invoke cancelAutomation, followed by immediate invocations to add new automations (scheduled for some time > currentTime)? For example, would this:

var currentTime = audioContext.currentTime; param.cancelAutomation(); param.linearRampToValueAtTime(v1, currentTime + t1); // where t1 is "suitably" large param.linearRampToValueAtTime(v2, currentTime + t2); // where t2 > t1// ... etc

just do the right thing? Or, would the API consumer need to perform additional synchronisation steps so as to avoid adding the new automations prior to the cancellation actually being performed by the audio thread?

For my use-case (see code sample above), the overall accuracy and timing of the resulting envelope is not as important as a click-and-pop free audio stream. As a result, the slope of the post-cancellation linear automation curve is not critical. I can choose t1 and v1 to give a suitably flat slope, with allowances made for the unpredictable time (and param value) of cancellation.

So for my particular application: the lack of control of the precise time of cancellation is okay, the lack of predictability of the time of cancellation is also okay. and the lack of knowledge of the value at the time of cancellation is okay.

I wonder how many other applications would find this API adequate, versus how many would find it inadequate?

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/344#issuecomment-151649265 .

[joeberkovitz]

@mark-buer As some additional background here, the group proposed something exactly like @cwilso's cancelAutomation() method (we called it something less likely to be confused with cancelScheduledValues, although I can't remember the name -- and I kind of like cancelAndHold). This was not a replacement for cancelScheduledValues, but an alternative.

The WG was on board with this, but at the last minute it was felt unnecessary based on the ability to capture the AudioParam's current value and then explicitly set that for the next render. But you made an important point about the possibility of a render occurring just after capturing the value, but before cancelling further automation and the subsequent call to setValueAtTime.

I think that point is the best argument for a cancelAndHold(), since there appears to be no way in the main control thread to atomically capture the current automation value and then immediately cancel any further automation, preventing any possible glitches.

[mrkishi]

Perhaps naively, I can't identify what complicates the implementation of a scheduled version of cancelAutomation/cancelAndHold.

It isn't an absolute necessity as I can invert the control flow, but you can see how it ends up much harder to reason about:

• User interaction > schedule automations > user interaction > cancel automations
• User interaction > wait > if no further interaction, schedule automations

I took a stab at implementing a proof of concept in plain Javascript, and the only thing I found that would block a native implementation is the current setValueCurveAtTime semantics (ie, the fact that you cannot setValueAtTime in the middle of a curve automation). Would anyone check it out?

I don't work on any WAA implementation, so I have a feeling I'm being incredibly dumb and missing a lot of details. If that gist is sound, though, what's the reasoning behind the "no overlapping automations with setValueCurveAt" rule?

ps. I had to keep track of the automation timeline in Javascript: this is nowhere near a complete implementation. There are several aspects that are currently unclear in the spec: I tried to follow Chrome (stable) where possible.

[rtoy]

On the way back from TPAC, I had a chance to work out some of the details. At time t0, assume we call cancelScheduledValues(tc). (tc >= t0, of course). Let t1 be the event just before tc and t2 be the event just after tc. If there is no event t1, then there's nothing to do. If there is no event t2, there's almost nothing to do, but we'll handle that case later.

Thus, we have events at time t1 and t2 and t1 < tc < t2. Then compute the value, vc, of the time line at time tc and insert a new event of the same type as t2 but with a new value of vc. Then remove all events whose time is greater than tc.

Thus, between t1 and tc, the timeline produces the same values as if cancelScheduledValues were not called, and, as usual, after time tc, the value of the timeline is held.

Our biggest issue was whether you could do this with an exponential ramp. The answer is yes, it works out to produce exactly the same curve as before. The linear ramp is obviously the same. For a value curve, I haven't worked out all the details, but I think it can be done by slightly modifying the original curve by adding one additional point at time tc with value vc and with a new setValueCurve duration of tc-t1. Since linear interpolation was used between sample points, the line segments are still the same.

And note that I did not specify whether t0 < t1 or not. With the insertion of the new event, and the curve being unchanged, it doesn't matter if the automation was on-going when cancelScheduledValues was called. The curve doesn't change shape and everything continues smoothly.

Since we insert an event, if new events are introduced between t1 and tc, everything would behave as expected as if the cancel were never there. Only at time tc would there be a change in the output. If new events are added after the call to cancelScheduledValues that occur after tc, they are processed as expected to because there is a real event with the desired value at time tc.

Now for the case where there is no event at t2. For any event at t1, there's nothing to do because the automation ended at t1 and would be held at that constant value. Except if that event is our good friend setTargetValueAtTime. I don't think we covered this case, but my expectation is that between t1 and tc, we ramp as would be expected and then hold the value constant after time tc. This will require "magic" in the implementation to keep track of things because we can't actually insert a setValue or anything because that would effectively cancel the setTarget.

This is somewhat complicated if a new event is scheduled between t1 and tc. The setTarget is effectively cancelled, so we would want to cancel the magic of holding the output. I think this is not difficult for implementations to handle correctly.

[rtoy]

Here's a proof that the replacement exponential produces the same results. Without loss of generality assume t0 = 0. Then the original curve is

v(t) = v0*(v1/v0)^(t/t1)

At time tc, we have v(tc) = v0*(v1/v0)^(tc/t1). The new curve, v'(t) starts at v0 and goes to vc:

v'(t) = v0*(vc/v0)^(t/tc)
      = v0*(1/v0)^(t/tc)*vc^(t/tc)

Substitute the value of vc:

v'(t) = v0*(1/v0)^(t/tc)*[v0*(v1/v0)^(tc/t1)]^(t/tc)
      = v0*(1/v0)^(t/tc)*v0^(t/tc)*(v1/v0)^((tc/t1)*(t/tc))
      = v0*(v1/v0)^(t/tc)

That is, v'(t) = v(t) for all t.

[rtoy]

Oh, canceling setTargetAtTime was specified in pull request #580. In light of the desire for sample and hold, this might need to be revisited.

[g200kg]

I am not sure because there are so many issues about canceling automation, the following simple ADSR use-case is solved already ?

KeyOn at t0 // t1=t0+attackTime gain.gain.setValueAtTime(0, t0); gain.gain.linearRampToValueAtTime(1, t1); // linear attack curve to 1 gain.gain.setTargetAtTime(s, t1, d); // decay curve to sustain level

KeyOff at t2 gain.gain.cancelScheduledValues(t2); gain.gain.setTargetAtTime(0, t2, r); // release curve to 0

Current implementation of Chrome almost work nicely but if KeyOff is occured during attack phase (t0 < t < t1), release curve can't be made. And I think it need external calculation of V0 of release curve.

This example is available at : http://g200kg.github.io/adsrtest/ https://github.com/g200kg/adsrtest

I think it may solved by 1) cancelScheduledValues can hold the last value. or 2) newly activated event use current automation value (not .value attribute) as V0. or 3) current automation value always reflect to .attribute value and newly activated event use that as V0.

[rtoy]

FWIW, I have a prototype implementation of cancel and hold. It basically works for as people would want, I think, but there are lots of corner cases that I've found that need to be specified.

[rtoy]

Here is an algorithm for how cancelScheduledValues can work to cancel and hold.

Assume we call cancelScheduledValues(tc), where tc is the cancellation time. Then

1. Let E1 be the event (if any) at time t1 where t1 is the largest number satisfying t1 ≤ tc.
2. Let E2 be the event (if any) at time t2 where t2 is the smallest number satisfying tc<t2.
3. If E2 exists:
   1. If E2 is a linear or exponential ramp,
      1. Rewrite E2 to be the same kind of ramp ending at time tc with an end value that would be the value of the original ramp at time tc.
      2. Go to step 5.
   2. Otherwise, go to step 4.
4. If E1 exists:
   1. If E1 is a setTarget,
      1. Implicitly insert at setValueAtTime at time tc with the value that the setTarget would have at time tc.
      2. Go to step 5.
   2. If E1 is a setValueCurve(c, t3, d)
      1. If tc>t3+d, go to step 5.
      2. Otherwise,
         1. effectively replace this event with setValueCurve(c,t3, tc-t3). However, this is not a true replacement; this automation must take care to produce the same output as the original, and not one computed using a different duration. (That would cause sampling at the wrong points.)
         2. Go to step 5.
      3. Otherwise, go to step 5.
5. Remove all events with time greater than tc.

It is undefined if new events are inserted between t1 and tc, after having called cancelScheduledValues(tc), but calling cancelScheduledValues again between t1 and tc is valid to cancel the cancel event that was previously scheduled at time tc.

[rtoy]

Here's a link to some notes that @hoch and I made: https://drive.google.com/open?id=1dsXZLd2TO0CY2iKLEQwyV_zdJ50wp76agtvOORla8ss

I tried to make it public, but I failed, so you'll need to request access. Unless you're part of the working group, I probably won't grant access. Next time, I'll create such documents outside of google where I can share public stuff more easily.

[rtoy]

Here's a public link to the pdf version: https://drive.google.com/file/d/0B9pZaKd286SkU083dEJzd2RpX2s/view?usp=sharing

[mark-buer]

The algorithm outlined in the PDF seems logical. The edge cases are certainly subtle.

An alternative (equivalent?) way of thinking about cancellation is as follows:

Let Timeline(t) be an array of automation events that take effect as of time t. Let Timelines be an array of Timeline. Let Timelines.length be the total number of Timeline within Timelines.

At context start, create Timeline(0) (containing an empty array of automation events), and add it to Timelines.

Each cancelScheduledValues(tc) invocation will cause the creation of a new Timeline(tc) which is added to the end of Timelines. No preexisting Timeline or automation event is modified.

Automation API methods will act upon the array of automation events within the Timeline given by Timelines[Timelines.length - 1]. Any attempt to add automations prior to the time t associated with that Timeline(t) can be detected and reported as an exception (or ignored, or be given intentionally undefined behaviour).

The audio renderer will examine Timelines[0 .. Timelines.length - 1] thus ensuring that the appropriate Timeline (and automation event) is used for each sample. Timelines will be tidied by removal of lapsed Timeline.

I'm not suggesting the above as an implementation strategy (although it might work), but as a (perhaps?) simple description of the observed behaviour of cancelScheduledValues.

I've probably missed an obvious reason why this isn't equivalent.

[rtoy]

Thanks for your comments. I'm a little confused on what Timeline(t) really means. Are these all the events scheduled at time t?

An alternative that I considered after writing the above proposal was to have cancelScheduledValues actually insert a new "cancel" event at time tc. This event would "remember" the event immediately following time tc, and remove all events after time tc. The timeline would progress as normal, including any insertions of new events. When it is time to process the cancel event, it basically replaces itself with a setValueAtTime event with the desired hold value, based on the previous event and the saved event. I didn't pursue this completely.

[joeberkovitz]

To avoid breaking changes name this as a new method cancelAndHoldAt(t).

[rtoy]

Maybe cancelAndHoldAtTime, since everything else is AtTime. Or even holdValueAtTime?

[padenot]

I'd prefer the former, because the fact that this removes the events is important.