Does AudioContext.latencyHint affect AudioWorkletProcessor::process Callback Interval?

[guest271314]

Describe the issue

Does AudioContext.latencyHint affect AudioWorkletProcessor::process Callback Interval?

If so that language and effect is not clearly specified.

Where Is It

• https://webaudio.github.io/web-audio-api/#dictionary-audiocontextoptions-members
• https://webaudio.github.io/web-audio-api/#processing-model-background
• https://webaudio.github.io/web-audio-api/#latency
• https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-instantiation
• https://heycam.github.io/webidl/#construct-a-callback-function

Additional Information

At DevTools open WebAudio tab. Observe the values of Callback Interval corresponding to different AudioContext.latencyHint settings

"playback" => Callback Interval 23.217 "interactive" => Callback Interval 11.626 "balanced" => Callback Interval 9.9 0.5 => Callback Interval 185.729

[guest271314]

Different AudioWorkletProcessor.process() callback intervals can be observed when setting different AudioContent.latencyHint values (https://plnkr.co/edit/qBUEPiBWZyvRxD65F1ws?p=preview), with "balanced" always having the least number of process() calls, yet inconsistent results when comaring 0.5, 1.0, and "playback".

Could not locate language in the current specification that unequivocally states that latencyHint SHOULD or MUST result in different callback intervals for process() function, nor that the results should be consistent relevant to the value set.

Is the result working as intended?

[rtoy]

It's not really clear how you compute the callback interval. But the latency value can affect when process() is called. This is also dependent on the sample rate of the audio as well as the operating system.

You might be interested in getOutputTimestamp. See also WebAudio/web-audio-api#12 and related issues.

[padenot]

It's a hint, by definition it is valid, for example, to implement it and do nothing. This is spelled out. The actual latency must however be reported, but doesn't have to match. it can be absolutely backward and still be compliant: an implementation cannot always control the actual latency, regardless of what is being asked by authors.

The timings at which process are called are not written in the spec, because a performant implementation doesn't control when it's being called, this is explained in section 2.1 (non normative of course).

[guest271314]

It's not really clear how you compute the callback interval.

Count the number of times process() is executed within 1 second with different valuesset for latencyHint.

There is an observable relationship between the value set at latencyHint and the number of times process() is executed within a given time span. However that relationship is not clearly specified (AFAICT not specified at all).

[guest271314]

The timings at which process are called are not written in the spec, because a performant implementation doesn't control when it's being called

Well, that appears to be what is occurring in the only implementation of AudioWorklet that am aware of (Chromium). It is currently possible to observe process() being executed N number of times directly corresponding the AudioContext.latencyHint.

[guest271314]

For a set of 10 tests, where the value is the number of process() calls per 1 second.

[
  [
    {
      "balanced": 303
    },
    {
      "interactive": 344
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 302
    },
    {
      "interactive": 347
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 311
    },
    {
      "interactive": 348
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 300
    },
    {
      "interactive": 344
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 308
    },
    {
      "interactive": 348
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 313
    },
    {
      "interactive": 340
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 305
    },
    {
      "interactive": 345
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 302
    },
    {
      "interactive": 348
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 312
    },
    {
      "interactive": 344
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ],
  [
    {
      "balanced": 308
    },
    {
      "interactive": 344
    },
    {
      "playback": 352
    },
    {
      "0.5": 384
    },
    {
      "1": 384
    }
  ]
]

Code to reproduce

main.js

class WorkletNode extends AudioWorkletNode {
  constructor(ctx) {
    super(ctx, "worklet-processor", {
      numberOfInputs: 1,
      numberOfOutputs: 1,
      channelCount: 1,
    });
  }
}

const latencyHints = [{
  latencyHint: "interactive"
}, {
  latencyHint: "balanced"
}, {
  latencyHint: "playback"
}, {
  latencyHint: 0.5
}, {
  latencyHint: 1
}];

async function main(
  latencyHint
) {
  const processorCallbackCount = [];
  const ctx = new AudioContext({
    sampleRate: 44100,
    latencyHint: latencyHint
  });
  console.log(ctx.baseLatency);
  const osc = new OscillatorNode(ctx, {
    type: "sine",
    frequency: 400
  });
  osc.start();
  await ctx.audioWorklet.addModule("./processor.js");
  const worklet = new WorkletNode(ctx);
  worklet.port.onmessage = e => processorCallbackCount.push(e.data);
  worklet.onprocessorerror = e => {
    console.error(e);
    console.trace();
  };
  osc.connect(worklet).connect(ctx.destination);
  await new Promise(resolve => setTimeout(resolve, 1000));
  osc.stop();
  worklet.port.close();
  osc.disconnect(worklet);
  worklet.disconnect();
  await ctx.suspend();
  await ctx.close();
  return processorCallbackCount.length;
}

async function getAudioWorkletProcessorLatency() {
  const result = [];
  for (const {
      latencyHint
    }
    of latencyHints) {
    result.push({
      [latencyHint]: await main(latencyHint)
    });
  }
  return result.sort((a, b) => Object.values(a)[0] < Object.values(b)[0] ? -1 : 0);
}

async function getAudioWorkletProcessorLatencyStatistics(length) {
  const result = [];
  for (const _ of Array.from({
      length
    })) {
    result.push(await getAudioWorkletProcessorLatency());
  }
  return result;
}

getAudioWorkletProcessorLatencyStatistics(10)
  .then(result => {
    const json = JSON.stringify(result, null, 2);
    console.dir(result);
    document.querySelector("pre").textContent = json;
  }, console.error);

main();

processor.js

class WorkletProcessor extends AudioWorkletProcessor {
  process(inputs, outputs, parameters) {
    // Output DC.
    //console.log(globalThis.setTimeout);
    //return;
    const input = inputs[0][0];
    const output = outputs[0][0];
    // console.log(output);
    output.set(input);
    this.port.postMessage({currentTime});
    return true;
  }
}

registerProcessor("worklet-processor", WorkletProcessor)

[rtoy]

Since you have chosen a sample rate of 44.1 kHz, there should be 44100/128 calls to process, or about 344 calls per sec. If you're not getting this, file a bug against Chrome.

BTW, I think your process method is more complicated than it needs to be and generates too much garbage. Just add a counter and increment it each time it's called. Inspect the currentTime and post a message when 1 sec (or 10 sec or whatever) have passed. No garbage generated.

Anyway, I think the spec is pretty clear: process gets called about 344 times per sec in this case. They may be bursty, but the average should be that.

[guest271314]

BTW, I think your process method is more complicated than it needs to be and generates too much garbage.

The code originates from a Chromium bug and is only used to demonstrate that AudioContext.latencyHint value has a concrete effect on the number of AudioWorkletProcessor.process() calls - though cannot locate any reference to such a symbiotic relationship between the two at the specification. Is that behaviour intended?

@rtoy The point of this issue is that different values set at AudioContext.latencyHint directly impact the number of times AudioWorkletProcessor.process() is executed.

From the responses to this issue thus far it is not clear that is the intended behaviour.

What is clear is that behaviour is not described in Web Audio API specification.

Therefore, the issue is not resolved by any definitive answer from contributors to the specification.

[rtoy]

The processing algorithm is described here: https://webaudio.github.io/web-audio-api/#processing-model. The latencyHint isn't involved.

Anyway this issue is closed. You've already filed a bug for Chrome. Further discussions will happen there. I don't think there's any inconsistency in the spec, but the processing algorithm is pretty complicated.