search

WebAudio / web-audio-api

The Web Audio API v1.0, developed by the W3C Audio WG
https://webaudio.github.io/web-audio-api/
Other
1.04k stars 165 forks source link

MediaStreamAudioSourceNode channelCount and "outputs" numberOfOutputs are incorrect #2496

Closed guest271314 closed 1 year ago

guest271314 commented 1 year ago

Describe the issue

channelCount and numberOfOutputs of MediaStreamAudioSourceNode are incorrect, wrong, on both Chromium 105 and Firefox 101.

Where Is It

1.24. The MediaStreamAudioSourceNode Interface

The number of channels of the output corresponds to the number of channels of the MediaStreamTrack.

Additional Information

What MDN says https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamAudioSourceNode:

Channel count | defined by the first audio MediaStreamTrack passed to the AudioContext.createMediaStreamSource() method that created it.

Test and verify:

1 channel sound file. 2 channel sound file. Set each to src of an <audio> element, in succession, e.g., 

$ mkvmerge -J ImperialMarch60.webm
  ...
  "file_name": "ImperialMarch60.webm",
  ...
  "tracks": [
    {
      "codec": "Opus",
      "id": 0,
      "properties": {
        "audio_bits_per_sample": 32,
        "audio_channels": 1,
        "audio_sampling_frequency": 48000,
        "codec_id": "A_OPUS"   
      ...
$ mkvmerge -J blade_runner.webm
  ...
  "file_name": "blade_runner.webm",
  ...
  "tracks": [
    {
      "codec": "Opus",
      "id": 1,
      "properties": {
        "audio_bits_per_sample": 32,
        "audio_channels": 2,
        "audio_sampling_frequency": 48000,
        "codec_id": "A_OPUS"
        ...

Use HTMLMediaElement.captureStream(), or on Firefox HTMLMediaElement.mozCaptureStream() to get a MediaStream from a media element. Pass the MediaStream to MediaStreamAudioSourceNode(). 

<!DOCTYPE html>

<html>
  <head> </head>

  <body>
    <audio autoplay controls src="./ImperialMarch60.webm"></audio>
    <script> 
      // ./blade_runner.webm
      const audio = document.querySelector('audio');
      let stream, track, settings;
      audio.onplay = async () => {
        stream = audio['mozCaptureStream' in audio ? 'mozCaptureStream' : 'captureStream']();
        [track] = stream.getAudioTracks();   
        settings = track.getSettings();
        const ac = new AudioContext();
        const msasn = new MediaStreamAudioSourceNode(ac, {mediaStream: stream});
        console.log(msasn);
        console.log(track.channelCount, settings, await track.getConstraints());
      };
    </script>
  </body>
</html>

Observe channelCount is always 2 and numberOfOutputs is always 1.

Provide any additional information

We already cannot rely on MediaStreamTrack getSettings() to get channelCount on eiher Firefox or Chrome for a track from captureStream(). We also cannot rely on Web Audio API channelCount to be true and correct.

Hence, we cannot get correct channel count to set at WebCodecs AudioEncoder.configure().

This must be reliable, or be removed.

guest271314 commented 1 year ago

The same thing happens for MediaElementAudioSourceNode. channelCount is 2 when the actual channel count is 1.

The language of the specific needs to use "MUST" re channelCount being accurate.

We can get accurate channel count using https://github.com/w3c/mediacapture-transform MediaStreamTrackProcessor, which demonstrates Web Audio API initialize channelCount, that evidently does not get updated, is incorrect, wrong; on both Firefox and Chrome.

<audio autoplay controls src="./ImperialMarch60.webm"></audio>
    <script>
      // blade_runner.webm
      const audio = document.querySelector('audio');
      let stream, track, settings, msasn;
      const ac = new AudioContext();
      audio.onloadedmetadata = async()=>{
        stream = audio['mozCaptureStream'in audio 
          ? 'mozCaptureStream' 
          : 'captureStream']();
        [track] = stream.getAudioTracks();
        settings = track.getSettings();
        msasn = new MediaStreamAudioSourceNode(ac,{
          mediaStream: stream,
          channelInterpretation: 'discrete',
          channelCountMode: 'explicit'
        });
        if ('MediaStreamTrackProcessor'in globalThis) {
          processor = new MediaStreamTrackProcessor({
            track
          });
          reader = processor.readable.getReader();
          console.log(
             track.channelCount, // undefined
             msasn.channelCount, // 2
             (await reader.read()).value.numberOfChannels // 1
          );
          try {
            await reader.cancel();
          } catch (e) {
            console.log(e);
          }
        }
      }
      ;
    </script>

Is channelCount being inhertied from AudioContext constructor, and never updated to reflect channel count of MediaStreamTrack?

A solution to this would be addition of getChannelCount() to get accurate current channel could of an AudioNode.

padenot commented 1 year ago

channelCount is for when something is connected to an AudioNode. MediaElementAudioSourceNode and MediaStreamAudioSourceNode don't have AudioNode inputs (i.e. you can't connect anything to them meaningully). channelCount is therefore of no use.

numberOfOutputs is the number of output ports of an AudioNode. MediaElementAudioSourceNode and MediaStreamAudioSourceNode only have a single output port, so it's always 1.

If you wish to determine the channel count of a media, an AudioWorkletProcessor can be used, it always works. Otherwise, out-of-band signals can be used.

guest271314 commented 1 year ago

channelCount is therefore of no use.

That is the point of this issue.

channelCount shouldn't be there (in the specification re MediaStreamAudioDestinationNode and MediaStreamAudioSourceNode) given they don't do anything, and the values are always incorrect.

Chrome only support 1 channel input for getUserMedia() and getDisplayMedia() so channelCount: 2 is always wrong.

padenot commented 1 year ago

It can't be removed because it's on the base class, that's how WebIDL works, and it's way too late to change this now.

guest271314 commented 1 year ago

If you wish to determine the channel count of a media, an AudioWorkletProcessor can be used, it always works.

An example of what you describe will be useful. As there is no way that I am aware of to get the channel count from captureStream() because getSettings() doesn't return channelCount.

It can't be removed

So we know we have a lame duck property and value that reports incorrect values, is useless though for legacy purposes cannot be remove?

guest271314 commented 1 year ago

See https://github.com/davedoesdev/webm-muxer.js/blob/375cc63f5bd940303037026347ff141b2ee2281f/media-element-demo.js#L62-L63

const audio_settings = audio_track.getSettings();
//audio_settings.channelCount etc are undefined!
guest271314 commented 1 year ago

https://github.com/davedoesdev/webm-muxer.js/issues/31#issuecomment-1186115719

Thanks, I can confirm channelCount is always 2, very helpful API that :)

Why continue to report incorrect values for channelCount of MediaStreamAudioSourceNode?

Surely we can do better than that.

guest271314 commented 1 year ago

To get 2 channels from getUserMedia() on Chromium to pass to MediaStreamAudioSourceNode we have to explicitly set echoCancellation to false and channelCount to 2, else the MediaStreamTrack of kind audio channelCount will be 1.