Always uses the channel count of the target sink rather than the source media, on pipewire-pulse

[pallaswept]

I have a Pipewire setup which disables channel upmixing, so that stereo streams receive proper treatment in stereo, and surround (5.1) streams receive proper treatment as surround sound. These streams are forwarded to 5.1 virtual sinks for said treatment.

In firefox, if I play a stereo source, it will open a 6 channel surround stream with the last 4 channels being silent. In other words, it decides to be surround even when it's not. The jack and alsa backends will change their stream channel count to match the media being played, so stereo is stereo and surround is surround.

[wtay]

PA_STREAM_NO_REMIX_CHANNELS would be the flag to use when connecting the playback stream. Otherwise pipewire pulse will upmix to the number of channels of the sink it links to.

[pallaswept]

PA_STREAM_NO_REMIX_CHANNELS would be the flag to use when connecting the playback stream. Otherwise pipewire pulse will upmix to the number of channels of the sink it links to.

I'm already preventing the upmixing in pipewire and it works fine, if I disable that, it will actually upmix.... the problem is that with upmixing disabled in pipewire, FF doesn't actually upmix to the number of channels of the destination sink, it just connects silent channels - ie if I play a stereo source, and connect to a sink with 6 channels, there will be 6 links, the first two carry FL and FR, and the other 4 are silent.

I take it from what documentation that I can find, that the PA_STREAM_NO_REMIX_CHANNELS flag needs to be implemented by the client, and can't be controlled by the user?

[wtay]

the problem is that with upmixing disabled in pipewire, FF doesn't actually upmix to the number of channels of the destination sink, it just connects silent channels

Probably not. upmixing is not usually done by applications. The way it usually works is that firefox says it has a stream with 2 channels and then the framework (pulseaudio or pipewire-pulse) upmixes it to the channels of the device.

The way this works in pipewire is that wireplumber reconfigures the application stream to the channel layout of the sink and then it proceeds connecting the channels one by one. If you disable upmix this effectively leaves the channels other than FL and FR silent. There is nothing the application can do about this usually.

PA_STREAM_FIX_CHANNELS will actually tell the application the final negotiated channel layout and can upmix if it wants.

PA_STREAM_NO_REMIX_CHANNELS tells wireplumber to leave the stream in the original channel layout, linking (in this case) FL and FR and leaving the other channels in the sink unlinked.

[pallaswept]

wireplumber reconfigures the application stream to the channel layout of the sink

It's never done that here (edit: Except for this bug in firefox, obviously). Using this config in the client conf file:

stream.properties = {
  stream.dont-remix        = true
  channelmix.disable       = true
  channelmix.upmix         = false
  ## only "psd", "none" or "simple" values are accepted
  channelmix.upmix-method  = none

Other applications that are told not to upmix in my pipewire config, only have the number of ports of the channel count of the actual source media, and they only link those channels. For example if I play the stereo track of this movie in haruna, it has a client node with 2 ports, and even when linking to a 5.1 surround sink, links FL and FR in haruna to FL and FR at the sink, and nothing else (C,LFE, RL and RR on the sink are simply not linked to), if I switch it to the 5.1 track of the movie, it will become (I understand it destroys the old node and creates a new one) a client node with 6 ports and link all 6 channels, if I switch back, the client node will have 2 channels and again only link those two channels to their matching ports on the sink. The client node always carries the channel count it's actually playing back. Firefox is an exception to this rule.

Edit: Correction - firefox when using its pulse backend is an exception to this rule. It works as described with either the JACK or ALSA backends.

[wtay]

The client node always carries the channel count it's actually playing back. Firefox is an exception to this rule.

So that would suggest firefox is using FIX_CHANNELS and then just leaves the channels empty?

[pallaswept]

The client node always carries the channel count it's actually playing back. Firefox is an exception to this rule.

So that would suggest firefox is using FIX_CHANNELS and then just leaves the channels empty?

Something like that? This is beyond my expertise on pulse. To perhaps give some insight, if I DO allow pipewire to upmix and remix, and then play back a stereo movie in FF into the 5.1 sink, it will still be a client with 6 ports, and it will not upmix, it will have 4 empty channels. With that same config using the haruna example above, I always have a client node with 6 channels, and there's always sound in all of them, even if I play the stereo soundtrack (because pipewire upmixes it). Pipewire doesn't upmix from firefox playing stereo because the channels are already there so it thinks it's already surround and doesn't need to be upmixed.

[wtay]

To perhaps give some insight, if I DO allow pipewire to upmix and remix, and then play back a stereo movie in FF into the 5.1 sink, it will still be a client with 6 ports, and it will not upmix, it will have 4 empty channels.

That sounds like FIX_CHANNELS behaviour and then not handling the extra channels.

[wtay]

This ultimately depends on what the user prefers. Some prefer automatic upmixing of stereo content, others prefer to not do any processing on the audio and leave the 4 extra channels empty.

It's also possible to add a firefox quirk to disable the fix flags, see pulse.fix.position here: https://docs.pipewire.org/page_module_protocol_pulse.html

[padenot]

Firefox can do a lot of things, in different ways, depending on the use case. I'm going to describe the intended behaviors, and then we can decide if it's not optimal.

If Firefox is playing regular media (e.g. stereo music, or a 7.1 movie or something like that, with an HTMLMediaElement), it will tell the layout and channel count of the media to the audio IO layer, and play that. That's for "regular" content that have a clear channel layout, explicit or implied (depending on the format).

If it's playing audio via the Web Audio API, the user of the Web Audio API can decide (in JavaScript) on the number of channel count to output. Depending on the OS and backend in use and also the number of channels, this can end up using all channels of a particular interface, and specifically use a "discrete" channel mapping, just writing the audio to the channels of that index. E.g. if the developer requests 5 channels, we don't really know what to do with it, so we write to the first 5 channels and pad with silence (again, highly backend specific). But if it's 6 it's probably 5.1. This is to support multichannel setup for art installations or research, that kind of thing: for example, 8 different speakers with a specific arrangement that is not 7.1 (usually all full-range speakers).

If this is not what is observed with a particular configuration, we can fix it. It would be ideal to clearly explain what is the configuration that seem to cause an issue, with a step by step explanation with details:

• pipewire config / routing etc.
• Firefox version
• Page being used to test
• any other details that seem relevant

since there's so many moving parts here that could influence what we experience.

We can also add an option to change between the two behaviour mentioned in https://github.com/mozilla/cubeb-pulse-rs/issues/86#issuecomment-1786635991, that's not particularly hard to do on our end, we have something to rematrix n-to-m channels somewhat intelligently.

[pallaswept]

Thanks so much for helping to figure out what's going wrong here @wtay and thanks for looking into it @padenot !

It would be ideal to clearly explain what is the configuration that seem to cause an issue, with a step by step explanation with details:

Firstly - if you want the short version of this, scroll down to the 'As a table' heading. You can read on for the details if you have the time.

pipewire config / routing etc. I'll do that last.... The details are not straightforward. Firefox version: Since at least 113 until 118.0.1 including nightly as of today...as long as I can remember. It's basically always been this way since I moved to Linux. Page being used to test: Various. Most commonly: Netflix, both with 5.1 and with 2.0 content, Stan (an Aussie streaming service) which is always stereo, Amazon Prime video, which is usually stereo but sometimes surround, Discord, which is both stereo and mono (voice is mono, media is stereo), Youtube, which is always stereo.

My configuration is complex and I'd be happy to share any of the config files so that you could test it out, but honestly most of that complexity is superfluous to this issue. I will summarize, and bold text the important parts (not yelling at you, just trying to help you skim read the important parts :) ) but you can skip to the TL:DR I'll put at the end of that summary (I'll make it a heading so you can find it easily), since as I said, most of this is superfluous to this issue - but you asked for my config, so here goes:

I have several virtual sinks, acting as mix busses (in order to be able to capture video with OBS Studio, with sound in different tracks; also to be able to set volumes appropriate to the scenario (like if I'm watching a car race and chatting with friends I might turn the race down so I can hear my friends)), all of these mix bus virtual sinks are 5.1 with standard surround layout, which feed into a single virtual sink, also 5.1, which then feeds out for monitoring, to one of various output devices, all of which are 5.1, and use virtual surround for stereo hardware such as headphones, although I may select to use the stereo devices directly as stereo devices without the virtual surround, depending on content (eg I don't assume to remix the likes of Vivaldi 😆 ). The virtual sinks which are mix busses, are fed by client apps which are directed by pipewire rules to link to their appropriate virtual sinks according to properties and metadata available; so the movies from firefox go to the Media Buss, firefox chat apps with webrtc metadata go to the chat buss, the audio from my phone's bluetooth goes to the phone busses, etc, and there is a 'catch all' which sends anything not specifically routed otherwise, to the 'Default Media Bus' sink. An image to give you the idea:

As you can see, all that's really important here is that we have audio clients being fed into a virtual sink with surround channel configuration, and that feeds into some output device.

You may have noticed that I have selected some of the channels coming from the three instances of firefox. The selected (grey) ones, actually have audio in them. The deselected (green) ones, are full of silence.

If I configure pipewire not to upmix or remix the audio from the clients (or anywhere else) - refer to the configuration snippet above - so that stereo stays stereo, and surround stays surround. An app (firefox excluded, which is the issue I'm reporting here) playing stereo content will have a client node with two output ports, an app playing surround content will have a client node with 6 output ports, and they will connect directly to the busses and follow the stream to my monitoring device of choice. Firefox, however, will ALWAYS have 6 output ports when connected to the sink with 6 input ports.

If I listen to media like music in stereo, it will play its two channels into the FL and FR of the mix buss sinks, there will be no link to the other four input ports of the sink, those channels will be silent and it will stay that way all the way to the monitoring device, so that way if I listen to a stereo device (say, headphones) then pipewire will only connect the first two channels (because I have disabled remixing) and I hear stereo in stereo, the silent channels are not linked. If I monitor on a surround device, it will link all 6 channels and the last 4 will be silent.

If I play surround media like a movie, it will link all 6 channels and if I monitor on a device which is surround (either natively or stereo hardware via virtual surround) then I get my surround in surround, if I listen on a device which is stereo, directly and not via virtual surround, I simply lack the extra 4 channels.

If I configure pipewire (as is the default) to allow upmixing and remixing of channels, then it is different, and this is probably the big issue for anyone using firefox and pipewire and surround hardware (which is apparently surprisingly rare or possibly unique to me - unlikely, but it seems that way given that I'm the only one reporting this). If I play surround media, then firefox will have a client node with 6 output ports, all containing audio, and they all link to the mix bus sinks, and surround audio is played all the way down the chain - again, if I monitor on surround hardware, I get surround sound, if I monitor on stereo hardware, pipewire will remix the audio into stereo by mixing all 6 channels into 2. _If I play stereo media, then firefox will still have a client node with 6 output nodes, and the last 4 will be filled with silence_. Accordingly, pipewire treats it as if it is surround sound, and does not perform any upmixing from stereo to surround, thus leaving 4 silent channels in a surround signal. If I monitor in stereo, the 'surround' signal is downmixed into stereo as before, thus ruining the actual stereo that it is playing by mixing it with silence. If I monitor in surround, I have 4 silent speakers.

TL;DR

• With default pipewire stream configuration: (the automagic approach)

Any other app will play stereo content as a stereo stream, get upmixed into a surround stream by pipewire, and play on surround on all 6 speakers,and play on stereo, downmixed into stereo. (works) Any other app will play surround content as a surround stream, play on surround on all 6 speakers, or play on stereo downmixed into stereo for the two speakers. (works)

Firefox will play stereo content as a surround stream with 4 silent channels, not get upmixed by pipewire, and play surround on only the first two speakers, leaving the other 4 silent. (broken), and play on stereo as surround downmixed into stereo, but the surround isn't really surround, it's stereo with 4 channels of silence now mixed into the stereo output (Broken) Firefox will play surround content as a surround stream, play surround on all 6 speakers, and downmix surround into stereo to play on two speakers. (works)

As a table:

Other apps:
Content:      Stream:       Surround Playback:          Stereo Playback:
Stereo        Stereo        Surround (Upmixed)          Stereo
Surround      Surround      Surround                    Stereo (Downmixed)

Firefox:
Content:      Stream:       Surround Playback:          Stereo Playback:
Stereo        Surround      Stereo (Broken-no upmix)    Stereo (Broken-downmixed with silence)
Surround      Surround      Surround                    Stereo (Downmixed)

• With pipewire configured not to upmix or remix streams: (the purist 'don't touch my audio' approach)

Any other app will play stereo content as a stereo stream, not get upmixed or remixed, stay a stereo stream, and play on surround as stereo, and on stereo as stereo. (works) Any other app will play surround content as a surround stream, not get remixed, play on surround as surround, and play on stereo with 4 missing channels. (works)

Firefox will play stereo content as a surround stream with 4 silent channels, not get upmixed or remixed, and play on surround as stereo, and on stereo as stereo. (works) Firefox will play surround content as a surround stream, and play on surround as surround, and play on stereo as surround with 4 missing channels. (works)

But then if you try to take the purist approach - which is not for everybody and there's no wonder why it's not the default - although it has the intended output, it's often not the most pleasant to listen to and doesn't take full advantage of all of your hardware - there comes a new problem - if you try to set rules to configure audio streams based on their channel count, so that you can say, pass stereo content through a filter for stereo content, and surround content through a filter for surround content, now it is broken again, because you can't distinguish stereo from surround streams by their channel count, because firefox always has 6 channels, even if it's stereo.

So, no matter which way you go, you run into problems with firefox's behaviour on pulse, where it always uses the channel count of the sink, rather than the channel count of the content, but always plays the channel count of the content, even if it doesn't match it's own channel count.

If you just disable pipewire-pulse, firefox will fall back to alsa and all these problems go away.... but then you miss out on lots of good stuff like stream metadata, so it's hardly a solution, but it is a good way to demonstrate how it ought to behave with pulse.

[pallaswept]

I tested this a little just now, to see if I could get any change in behaviour, maybe using the pulse.fix.position params, or something along those lines, and cutting out as much complexity as possible. I disabled all my routing, bypassed all the virtual sinks, and just used the default sink, selecting between a stereo HDMI output and a surround HDMI output on the same card. I enabled all of pipewire's usual upmixing and remixing, as per the defaults.

If I played a stereo track from haruna into the surround sink, it would have 6 output ports and link all six of them and all six carried audio, courtesy of pipewire's upmixing. If I changed default device, on the fly, the haruna node would change to have two output ports and link them both, stereo to stereo, all normal behaviour If I played a stereo source (youtube video) from FF, it would have two output ports and link them both to the stereo HDMI ports, all normal. If I then changed default device to the surround TV, youtube would have 6 output ports, and link them all, and the last 4 had no audio. No upmixing was performaed, as it was with haruna playing stereo into the surround TV

I tried all manner of different combinations of pipewire's pulse channel manipulation, fix.channels, default.channels, both globally, and by rules applied only to firefox, nothing I could do would change it's behaviour. No matter what, if firefox played stereo into a surround sink, it played 2 channels of audio and 4 of silence.

[padenot]

What if you start playback on the 6-channels output device in Firefox without switching on the fly?

[pallaswept]

That's the normal thing I'd do, and it's the same behaviour, ie client with 6 output ports, 4 filled with silence if the content is stereo.

[pallaswept]

client with 6 output ports, 4 filled with silence if the content is stereo.

I'm just wondering if there's been any progress on this?

I'm very surprised there's not more discussion of it, I guess the surround sound users of firefox (there are dozens of us!) just gave up and went to chrome. I'd rather not, though.... lil' help?