channelCount set at MediaStreamAudioDestinationNode MUST set MediaStreamTrack channelCount

[guest271314]

Describe the issue

Setting channelCount to 1 at MediaStreamAudioDestinatioNode constructor has no effect, or is not being recognized at MediaStreamTrack.getSettings().channelCount value - until MediaStreamTrackProcessor.readable.read() is called where input is an audio node with 1 channel.

applyConstraints({channelCount: 1}) throws, however the channelCount is dynamically changed at some point to 1.

Firefox returns empty JavaScript plain objects for getSettings() and getConstraints().

var ac = new AudioContext();
// 'max', 'clamped-max', 'explicit' `channelCountMode` have same effect; 'speakers' and 'discrete' `channelInterpretation` have same effect
var msd = new MediaStreamAudioDestinationNode(ac, {channelCount: 1, channelCountMode: 'explicit', channelInterpretation: 'discrete'});
var source = new MediaStreamAudioSourceNode(ac, {mediaStream: msd.stream});
var [track] = msd.stream.getAudioTracks();
var settings = track.getSettings();
console.assert(settings.channelCount === 1, [settings.channelCount]); // Assertion failed: [2]
track.applyConstraints({channelCount: 1})
.then(() => {
  // MediaStreamTrack contraints: {}
  console.log('MediaStreamTrack constraints applied:', track.getConstraints());
})
.catch((e) => {
  // MediaStreamTrack constraints not applied: OverconstrainedError Cannot satisfy constraints
  console.error('MediaStreamTrack constraints not applied:', e.name, e.message); 
  return track.getConstraints();
})
.then((constraints) => {
  console.log('MediaStreamTrack contraints:', constraints); // 
  if ('MediaStreamTrackProcessor' in globalThis) {
    var osc = new OscillatorNode(ac);
    osc.connect(msd);
    // MediaStreamTrack.channelCount before starting OscillatorNode: 2
    console.log('MediaStreamTrack.channelCount before starting OscillatorNode:', track.getSettings().channelCount); // 2
    osc.start();
    // MediaStreamTrack.channelCount before starting OscillatorNode: 2
    console.log('MediaStreamTrack.channelCount after starting OscillatorNode:', track.getSettings().channelCount); // 2
    var processor = new MediaStreamTrackProcessor(track);
    var {readable} = processor;
    var reader = readable.getReader();
    reader.read().then(({value, done}) => {
      // assertion does not fail, AudioBuffer.numberOfChannels is 1, MediaStreamTrack.channelCount is now 1
      console.assert(value.buffer.numberOfChannels === track.getSettings().channelCount, [value.buffer.numberOfChannels, track.getSettings().channelCount, value.buffer, track.getSettings()]);
      // assertion fails AudioBuffer.numberOfChannels is 1, MediaStreamTrack.channelCount is now 1
      // how, why, and when did MediaStreamTrack channelCount setting, constraint from MediaStreamAudioDestinatioNode stream change?
      console.assert(value.buffer.numberOfChannels !== track.getSettings().channelCount
      , JSON.stringify([
        value.buffer.numberOfChannels
        , track.getSettings().channelCount
        , source.mediaStream.getAudioTracks()[0].getSettings().channelCount
        ], null, 2)
      ); 
      /*
        AudioBuffer from AudioFrame numberOfChannels is 1
        MediaStreamTrack from MediaStreamAudioDestinationNode channelCount is now 1
        MediaStreamTrack from MediaStreamAudioSourceNode channelCount is now 1
        No constraints have been set 

        Assertion failed: [
         1,
         1,
         1
        ]
      */
      osc.stop();
      track.stop();
      reader.releaseLock();
      return readable.cancel().then(() => ac.close());
    })
    .catch(console.trace);
  }
});

Where Is It

MediaStreamAudioDestinationNode

channelCount | 2

MediaStreamAudioSourceNode

After construction, any change to the MediaStream that was passed to the constructor do not affect the underlying output of this AudioNode.

How can this be true when applyConstraints() is used?

This is not true in the following code where the channelCount is expected to be set to 1, is not set to 1 by MediaStreamAudioDestinationNode constructor options and reading a frame from MediaStreamTrackProcessor.readable.read()

channelCount

channelCount, of type unsigned long

channelCount is the number of channels used when up-mixing and down-mixing connections to any inputs to the node. The default value is 2 except for specific nodes where its value is specially determined. This attribute has no effect for nodes with no inputs. If this value is set to zero or to a value greater than the implementation’s maximum number of channels the implementation MUST throw a NotSupportedError exception.

In addition, some nodes have additional channelCount constraints on the possible values for the channel count:

MediaStreamAudioDestinationNode is not listed as having known restrictions on the channelCount value being set and returned.

For the above reasons changes to the specification

• channelCount set at MediaStreamAudioDestinationNode constructor MUST set MediaStreamTrack channelCount constraint
• channelCount set at MediaStreamAudioDestinationNode constructor MUST be reflected at MediaStreamAudioDestinationNode.MediaStreamTrack.getSettings()
• MediaStreamAudioDestinationNode.MediaStreamTrack.applyConstraints({channelCount: channelCount}) MUST apply the constraint, once the constraint is applied and Promise fulfilled be reflected at MediaStreamAudioDestinationNode.MediaStreamTrack.getConstraints()

The above changes will ensure that the channelCount set on a MediaStreamAudioDestinationNode by user is set by implementations on the MediaStreamTrack, observable via the appropriate methods that provide statistics and at the actual MediaStreamTrack channel count settings.

Additional Information

• https://github.com/w3c/webrtc-extensions/issues/63
• https://github.com/w3c/mediacapture-main/issues/775

[guest271314]

When {sampleRate: 22050} is set at AudioContext constructor the result can be inconsistent with regard to MediaStreamTrack.getSettings().

Without setting a sampleRate

Assertion failed: [2]
MediaStreamTrack constraints not applied: OverconstrainedError Cannot satisfy constraints
MediaStreamTrack contraints: {}
MediaStreamTrack.channelCount before starting OscillatorNode: 2
MediaStreamTrack.channelCount after starting OscillatorNode: 2
 Assertion failed: [
  1,
  1,
  1
]

compare first run with {sampleRate: 22050}

Assertion failed: [2]
MediaStreamTrack constraints not applied: OverconstrainedError Cannot satisfy constraints
MediaStreamTrack contraints: {}
MediaStreamTrack.channelCount before starting OscillatorNode: 2
MediaStreamTrack.channelCount after starting OscillatorNode: 1
Assertion failed: [
  1,
  1,
  1
]

run again at console with {sampleRate: 22050} without reloading the document

Assertion failed: [2]
MediaStreamTrack constraints not applied: OverconstrainedError Cannot satisfy constraints
MediaStreamTrack contraints: {}
MediaStreamTrack.channelCount before starting OscillatorNode: 2
MediaStreamTrack.channelCount after starting OscillatorNode: 2
Assertion failed: [
  1,
  1,
  1
]

[guest271314]

Part of the issue is incorrect and inconsistent return value of MediaStreamTrack.getSettings() (Firefox returns empty objects for getSettings()), which when MediaStreamAudioDestinationNode and OscillatorNode which connects to the former are both constructed with {channelCount: 1} option should result in MediaStramTrack.getSettings().channelCount1 not ever being 2, however, in practice that is not the case (if an application is relying on the initial value of getSettings().channelCount to be correct and consistent).

<!DOCTYPE html>

<html>
  <head>
    <title>
      MediaStreamTrack.getSettings() returns incorrect, inconsistent
      channelCount
    </title>
  </head>

  <body>
    <h1>Click several times, observe output at console</h1>
    <script>
      document.querySelector('h1').onclick = async () => {
        const ac = new AudioContext();
        const msd = new MediaStreamAudioDestinationNode(ac, {
          channelCount: 1,
          channelCountMode: 'explicit',
          channelInterpretation: 'discrete',
        });
        const osc = new OscillatorNode(ac, {
          channelCount: 1,
          channelCountMode: 'explicit',
          channelInterpretation: 'discrete',
        });
        osc.connect(msd);
        osc.start(ac.currentTime);
        console.log(
          'console.log',
          osc.channelCount,
          msd.stream.getAudioTracks()[0].getSettings().channelCount
        ); // 1 2, should be 1 1
        setTimeout(
          () =>
            console.log(
              'console.log in setTimeout()',
              osc.channelCount,
              msd.stream.getAudioTracks()[0].getSettings().channelCount
            ),
          0
        ); // could be 1 1 or 1 2
        await scheduler.postTask(() =>
          console.log(
            'console.log in scheduler.postTask()',
            osc.channelCount,
            msd.stream.getAudioTracks()[0].getSettings().channelCount
          )
        ); // could be 1 1 or 1 2
        await ac.close();
      };
    </script>
  </body>
</html>

Chromium bug https://bugs.chromium.org/p/chromium/issues/detail?id=1193248

[guest271314]

[rtoy]

The specification for how channel count and mode and interpretation work is unclear. I don't actually know what the expectations are.

[guest271314]

The plain language channelCount strongly implies that the user can directly set channel count for the underlying MediaStreamTrack. 'explicit' cannot means anything other than that, relevant to the channelCount (I am not sure what else 'explicit' could be refering to other than channelCount)

determines how channels will be counted when up-mixing and down-mixing connections to any inputs to the node

here we are "down-mixing" the evidently default 2 channel to 1 channel, explicitly.

is the exact value as specified by the channelCount

Re channelInterpretation

Up-mix by filling channels until they run out then zero out remaining channels. Down-mix by filling as many channels as possible, then dropping remaining channels.

by setting 'discrete' I am expecting that if, by some undisclosed, or known internal algorithm, if the MediaStreamTrack still has 2 channels after the above that all except for 1 channel, the API runs the "dropping remaining channels" part of the algortithm(s).

[padenot]

WG Discussion:

Good catch, there lots of unknowns in the area:

• Is changing the channelCount of a MediaStreamTrack that is not a microphone intended? What should be the result? This is in the MediaCapture and Streams: https://w3c.github.io/mediacapture-main/. For a mic, it's useful to force mono for stereo mic (or the opposite), for example, but for others, I'm not sure. The Web Audio API itself can downmix or upmix and that's well defined.
• When this changes, should we change the channel count of our objects that wrap those MediaStream{,Track}s
• Same issue comes from HTMLMediaElement with a number of channels in the media file that changes, when used with MediaElementAudioSourceNode.

[guest271314]

• Is changing the channelCount of a MediaStreamTrack that is not a microphone intended? What should be the result? This is in the MediaCapture and Streams: https://w3c.github.io/mediacapture-main/. For a mic, it's useful to force mono for stereo mic (or the opposite), for example, but for others, I'm not sure. The Web Audio API itself can downmix or upmix and that's well defined.

Yes. It does not matter what the underlying source of the track is.

With regards to actual audio input or API initially used to get the track. For example, when we pass mediaStream from getUserMedia() to MediaStreamAudioSourceNode we must be able to set constraints on the track to whatever we want.

If the Web Audio API can down-mix then that is what must consistently occur when we set channelCount to 1 at an audio node constructor, or when using applyConstraints({channelCount: 1}), however that throws OverConstrained error at both Chromium and Firefox.

• When this changes, should we change the channel count of our objects that wrap those MediaStream{,Track}s

Yes.

• Same issue comes from HTMLMediaElement with a number of channels in the media file that changes, when used with MediaElementAudioSourceNode.

Yes. Change the implementation MUST change the channel count internally, and those changes MUST be observable at getSettings() and getConstraints() and other methods that return settings, and constraints of the MediaStreamTrack, else all of those options and constraint setting and gettings methods are, as evident, inconsistent and incorrect, and useless.

[padenot]

AudioWG virtual F2F:

• Yes, let's do that.

[guest271314]

@padenot See https://github.com/w3c/mediacapture-main/issues/775.

[hoch]

TPAC 2022: So the problem is that the result of query on MediaStreamTrack.getSettings().channelCount value does not match the channel count of MediaStreamAudioDestinationNode.

This should be handled by the construction of MediaStreamTrack: https://w3c.github.io/mediacapture-main/#dfn-create-a-mediastreamtrack

If you see problems in getting a correct value, please consider file an issue against browsers.