Mixer should better handle mixing input sources

[SingingTree]

I've encountered some issues with using the mixer in the WASAPI backend to handle input streams. Starting a discussion to bounce some ideas and confirm that my concerns are not a product of misunderstanding the mixing API.

• The mixer header specifies many signatures that take a stream params and a mixer params. However the usage of these in the WASAPI backend is odd to me: there are times when a the stream params are passed where I would expect the mixer params and vice versa. I suspect this usage is intended and these arguments represent something more along the lines of input and output params. However, the naming conventions here are unintuitive to me. If this is intended I would prefer the signatures exposed something more along the lines of mixer_input_params and mixer_output_params.
• The current impl of cubeb_should_mix gates on if the stream (input?) doesn't have a defined layout. This is breaking for me when dealing with WASAPI input devices with more than 2 channels (the loop back case is what I'm trying to deal with) because the check here fails due to layout not being detect. The layout needing to be defined feels like a sane constraint, right? In which case it seems like the WASAPI backend will need to do better detection for inputs such that we do not have undefined layouts.
  • The case where our input and outputs have different channels but we don't mix due to undefined layout seems potentially nasty. We should at least log here, should we fail harder?
• Do we have any idea on how often we encounter > stereo layout? I'm running into issue with a MacBook Pro 2016 giving me a 2F2 layout under windows.
• Do we need special handling for different layouts? I see we have some for 3F2, but otherwise end up using mix_remap. My current plan is to get this working for my loopback use case. Any concerns here I should be aware of?

Would appreciate thoughts. @kinetiknz, @ChunMinChang

[SingingTree]

I've hacked on https://github.com/kinetiknz/cubeb/issues/383 and it appears to be the primary issue killing my input mixing and appears to be why layout is not detected in my case. Still interested in thoughts on the above.

[ChunMinChang]

• I think we should rename all parameters of the mixer by only using something prefixed by input and output. It will be clearer.

           +---------+
  input --->|  mixer  |---> output
           +---------+ 

• For undefined layout with different channels, we should downmix or upmix them by their channel counts. downmix_fallback will pass the data from input to output as much as it can and drop the extra channel. cubeb_upmix will copy through all the data from input to output and put silence in remaining channels.
  • e.g., if there are 3 input channels but only 2 output channels, then the 3rd input channel will be dropped.
  • e.g., if there are only 3 input channels but 4 output channels, then the 4th output channel will be all 0.
• We intended to add telemetry to get these information(bug 1287672), but the task is postponed. There are some unsolved problems if we need to continue this work:
  • The collected layout data is separated from the collected platform info. That is, we cannot know the 5.1 layout is from OSX or Windows.
  • The layout information is not added when metadata is demuxed(bug 1339362). All media will be transformed into SMPTE format before passing to cubeb. We should have a way to know the layout information of the media source.
• The default choice now is to use mix_remap. Some mixing conversions are defined in ITU specs. ITU-R BS.775-3 provides spec for downmixing 3F2(audio 5.1) data to 1F, 2F, 3F, 2F1, 3F1, and 2F2(downmix_3f2). I guess there is a spec for audio 7.1. We should check if there is a spec for the conversion you want before implementing our own version.