Inconsistant channel assignement with Jackd

[jdlvfr]

Sometimes when starting Jackd channel 1 becomes channel 7, sound moves form the fisrt rca jack to the seventh one. This occurs roughly 1 time out of 10. Restarting Jackd returns channel 1 to rca 1. I do the same test without Jackd, with sox and channel 1 stays on rca jack 1.

Configuration has been done with the deb package so last kernel, firmware and official asoundrc.

[flatmax]

How have you configured your jackd ? How do you run it ?

[jdlvfr]

Jackd was started with : jackd -dalsa -r44100 -n2 -p2048. Raspberry runs Raspbian lite, headless. Jackd was Jackd1, Jackd2 doesn't work in a headless setup because of Dbus. Whether the channel assignement is right or wrong, the sound stops after a few minutes and Jackd outputs a lot of Xruns. Small values, less than 100µs.

Then I tried to use playback only, no capture channels at all. Jackd was started with : jackd -dalsa -Phw:0 -r44100 -n2 -p2048. With playback only, lots of problems goes away, channel assignment seem correct and sound doesn't stops no more. Jackd still reports Xrun but now high values 5ms to 100ms.

Finally I compiled Jackd2 without dbus, full duplex gives the same inconsistent channel assignment but no more Xruns and playback only seem to work perfectly!

[robiwano]

Yes, I've also noticed these problems, when playing to Channel 1+2 and audio coming out on 7+8 (or something similarly unexpected).

[flatmax]

OK - I am not sure how dbus is used, nor how jackd uses it. Is it possible to post instructions on how you compiled jackd without dbus ? thanks

[robiwano]

You can follow the instructions here: http://forum.audioinjector.net/viewtopic.php?f=5&t=2727&start=30#p5749

[nettings]

Bumping this one more time. :) I'm using JACK and I've never seen the Octo come up with shuffled channels after a clean reboot. My systemd takes care of starting JACK once, and then it keeps running, sometimes for weeks, without problems. The only way I've found to force channel swapping is to repeatedly killall -ABRT jackd (or any other signal, I'm just using ABRT because systemd is configured to not restart after a KILL or TERM). Then, about 25% of the time, channels are swapped around. I only ever saw Out 1 move to Out 7, and so on in sequence, haven't looked at how the inputs get shuffled, will check that asap.

[flatmax]

OK - I have been thinking about this. Here are my thoughts ...

When the audio application is closed, the audio clocks are still running. These clocks need to settle (turn off) before soundcard operation starts again. If they don't settle then there is the possiblilty of channel swapping. Channel swapping is not normally a problem, however I have heard reports of it from different people and it is normally related to a quick shutdown/start up of an audio application.

One solution which may work is to rmmod the sound modules and modprobe them again (unload/reload them). It takes a small amount of time when scripted, however I am hoping it will ensure channel swapping doesn't happen when clocks don't get time to settle.

[nettings]

Unfortunately, that doesn't always solve it. I'm using a restart delay in systemd on the order of 10s (but even 30 don't always prevent it from coming up with swapped channels). I think at some point I saw swapped channels even after rmmoding and modprobing the modules, but I never ran that one in a scripted stress test. Will try that later today. But I seem to remember the only thing to fix it 100% is a clean reboot.

Does snd_soc_bcm2835_i2s also have an influence on i2s timing? Because it wasn't part of your workaround. Maybe including that one in the rmmod/modprobe cycle would help?

For the record, here's my module stack:

$ lsmod | grep snd_soc
snd_soc_cs42xx8_i2c    16384  1
snd_soc_cs42xx8        24576  1 snd_soc_cs42xx8_i2c
snd_soc_bcm2835_i2s    20480  2
snd_soc_audioinjector_octo_soundcard    16384  0
snd_soc_core          192512  3 snd_soc_audioinjector_octo_soundcard,snd_soc_cs42xx8,snd_soc_bcm2835_i2s
snd_compress           20480  1 snd_soc_core
snd_pcm_dmaengine      16384  1 snd_soc_core
snd_pcm               102400  6 snd_pcm_dmaengine,snd_soc_audioinjector_octo_soundcard,snd_soc_bcm2835_i2s,snd_soc_core
snd                    73728  8 snd_compress,snd_timer,snd_soc_core,snd_pcm

[flatmax]

ALSA core does have influence on when the audio clocks are started and stopped - which is where I think this problem starts. To debug this problem it requires a cro on the i2s clock lines, confirming that when the i2s clock lines keep running between audio apps runs, the problem occurs. when the i2s clock lines stop running between audio app runs, the problem doesn't appear.

When I get more time I will look into this.

On 24/4/19 7:19 pm, Jörn Nettingsmeier wrote:

Unfortunately, that doesn't always solve it. I'm using a restart delay in systemd on the order of 10s (but even 30 don't always prevent it from coming up with swapped channels). I think at some point I saw swapped channels even after rmmoding and modprobing the modules, but I never ran that one in a scripted stress test. Will try that later today. But I seem to remember the only thing to fix it 100% is a clean reboot.

Does snd_soc_bcm2835_i2s also have an influence on i2s timing? Because it wasn't part of your workaround. Maybe including that one in the rmmod/modprobe cycle would help?

For the record, here's my module stack:

|$ lsmod | grep snd_soc snd_soc_cs42xx8_i2c 16384 1 snd_soc_cs42xx8 24576 1 snd_soc_cs42xx8_i2c snd_soc_bcm2835_i2s 20480 2 snd_soc_audioinjector_octo_soundcard 16384 0 snd_soc_core 192512 3 snd_soc_audioinjector_octo_soundcard,snd_soc_cs42xx8,snd_soc_bcm2835_i2s snd_compress 20480 1 snd_soc_core snd_pcm_dmaengine 16384 1 snd_soc_core snd_pcm 102400 6 snd_pcm_dmaengine,snd_soc_audioinjector_octo_soundcard,snd_soc_bcm2835_i2s,snd_soc_core snd 73728 8 snd_compress,snd_timer,snd_soc_core,snd_pcm |

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Audio-Injector/Octo/issues/8#issuecomment-486140385, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFLUB35TUKV5RYQVYTZ6OTPSAQ3DANCNFSM4E2YVQBA.

[nettings]

Running the system into very high load also sometimes produce "kernel: bcm2835-i2s fe203000.i2s: I2S SYNC error!", and when this happens, the channels are almost always shuffled. Currently, I'm doing a make -j4 kernel build on a Pi 4 while running jackd and resampling 8 audio channels coming in from the network. Saw 2 shuffles during that time. For the record: there was no undervoltage or thermal throttling, as shown by vcgencmd get_throttled: throttled=0x0.

[nettings]

Correction: they are not "almost always" reshuffled. After seeing around 20 more sync errors (which always seem to occur in multiples of 2), it turns out that the probability of a shuffle happening is less than .5. I wonder: there are 3 ways of getting the stereo streams assigned wrongly, and one correct way. Yet nobody ever reported channel swaps other than 1/2 to 7/8 or vice versa. So I'm not sure I understand the underlying error mechanism. What prevents 1/2 to be shuffled to 3/4 or 5/6?

(Btw, in my current tests I'm only considering the analog outputs, but I'm assuming the inputs will do the same...)

[nettings]

Ok, I have a beautiful test case trying to watch a full HD movie on a PI4 with KODI feeding into the alsa-jack plugin into a jack dsp chain with some EQs and stuff. "I2S SYNC ERROR"s happening several times per minute, usually with a channel shuffle. On a "macro" scale, the system is not overloaded (CPU never over 50% on any single core), but of course you can never be sure about those very short-time congestions, and KODI opens a metric shitload of threads (> 20!) Now I wonder: if only I knew which kernel thread is handling the I2S SOC driver, I could increase its scheduling priority selectively (that used to help with firewire devices back in the days: http://subversion.ffado.org/wiki/IrqPriorities). The audio threads are running SCHED_FIFO with priority 40, i.e. below the 50 of the lowest kernel worker threads. But then the kernel also has non-SCHED_FIFO threads, and i wonder if one of those might be in charge of dealing with the I2S interrupts and be starved of CPU time...

[flatmax]

On 15/4/20 3:42 am, Jörn Nettingsmeier wrote:

Ok, I have a beautiful test case trying to watch a full HD movie on a PI4 with KODI feeding into the alsa-jack plugin into a jack dsp chain with some EQs and stuff. "I2S SYNC ERROR"s happening several times per minute, usually with a channel shuffle. On a "macro" scale, the system is not overloaded (CPU never over 50% on any single core), but of course you can never be sure about those very short-time congestions, and KODI opens a metric shitload of threads (> 20!) Now I wonder: if only I knew which kernel thread is handling the I2S SOC driver, I could increase its scheduling priority selectively (that used to help with firewire devices back in the days: http://subversion.ffado.org/wiki/IrqPriorities). The audio threads are running SCHED_FIFO with priority 40, i.e. below the 50 of the lowest kernel worker threads. But then the kernel also has non-SCHED_FIFO threads, and i wonder if one of those might be in charge of dealing with the I2S interrupts and be starved of CPU time...

Yes I would expect channel shuffling on active ALSA streams when there are xruns. Is it possible to increase the buffer size ALSA uses ? I would imagine that up to 10 ms or 15 ms would be acceptable for video sync. Alternatively you could increase buffer sizes even more and delay the video.

========= Our latest product launching now on Kickstarter : https://bit.ly/2Qr1557 Checkout the community email list : https://lists.audioinjector.net/mailman/listinfo/people

[nettings]

I'm using 256 frames per period, two buffers per period in jackd. The point of this isn't really to find a workaround for a usable system (i have that), but to drill down into the problem and prevent it from happening in the first place, since there is no way to recover from it at runtime short of tearing down the entire jack graph, reloading the modules, and building it back up again (that takes almost 30s in total for a graph this complex, and so is not an acceptable UX, plus the codec produces crackles when it restarts while amps are on). I'm trying to figure out about interrupt handlers on the Pi and experimenting with selectively increasing their scheduling priority - nothing in userspace should really be able to prevent an i2s interrupt handler from finishing its job, ever, and I wonder if that can be achieved... will report back. Re xruns: a "normal" userspace xrun in JACK does not cause any problems, it's not even audible for the most part. It's only the "I2S SYNC ERROR", i.e. a hardware hiccup, that shuffles channels with a certain likelihood.

[robiwano]

Max, how is the I2S sync managed? I suspect you run I2S at 192kHz/32bits stereo, then rebake it into 48 kHz,32bits 8 channels (via FPGA + driver)? Is there some flag in the "first" stereo pair to denote it as "first"?

[nettings]

That is precisely the problem: there isn't. You start the clocks, and you keep the hell counting. If you lose track (i.e. you see an I²S SYNC ERROR), it's happy channel lottery.

[robiwano]

Yes, I suspect so. Anyway, that's how I'd improve this, by using only 24 bits of audio in each sample frame, and use the other 8 bits as SYNC bits. This way, the FPGA and driver would be in 100% sync all the time.

[nettings]

I'm currently playing around with that as well. If you control the data (because you have an FPGA or DSP on the other side), you could add a sync signal to the data, but it would probably be enough to sacrifice only the LSB (also, I guess the wire data format is 24bits). So you lose 6dB S/N, but you still have 138 dB, that's plenty. The question is how to design the sync noise so that it doesn't hurt. Just setting the LSB to 1 on channel one and to 0 everywhere else might already be an improvement, but I wonder if that slight DC offset could be avoided. Another angle of attack is to try and get an external TDM sync clock pulse from the external FPGA or DSP via a GPIO (which the hardware doesn't support natively), but I'm a total newbie re kernel development and just getting my feet wet...

[flatmax]

This is a good idea.

However for the input channels there is already a sync signal. There are two channels which are always a constant, as they don't have audio data in them. I have written some code to detect and resynchronise the input channels.

This code here will detect the two input channels out of 8 which aren't in use. These are a good synch strategy for the input :

https://github.com/flatmax/gtkiostream/blob/ac3370ea45cc60da8e73b410e402c3590dba36d2/test/ALSAFullDuplexMinScan.C#L50-L60

Then this code will shift the input audio to be in the correct column (channel) alignment :

https://github.com/flatmax/gtkiostream/blob/ac3370ea45cc60da8e73b410e402c3590dba36d2/test/ALSAFullDuplexMinScan.C#L64-L72

It would be interesting to see if the same malalignment applies to the output data which is found on the input data.

On 16/4/20 12:38 am, Jörn Nettingsmeier wrote:

I'm currently playing around with that as well. If you control the data (because you have an FPGA or DSP on the other side), you could add a sync signal to the data, but it would probably be enough to sacrifice only the LSB (also, I guess the wire data format is 24bits). So you lose 6dB S/N, but you still have 138 dB, that's plenty. The question is how to design the sync noise so that it doesn't hurt. Just setting the LSB to 1 on channel one and to 0 everywhere else might already be an improvement, but I wonder if that slight DC offset could be avoided. Another angle of attack is to try and get an external TDM sync clock pulse from the external FPGA or DSP via a GPIO (which the hardware doesn't support natively), but I'm a total newbie re kernel development and just getting my feet wet...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Audio-Injector/Octo/issues/8#issuecomment-614079294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFLUB4DTFAVBJZ2YSXLPMTRMXBH7ANCNFSM4E2YVQBA.

[robiwano]

The question is how to design the sync noise so that it doesn't hurt. Just setting the LSB to 1 on channel one and to 0 everywhere else might already be an improvement, but I wonder if that slight DC offset could be avoided.

Since the CODEC itself only uses 24 bits, the low 8 bits of the I2S data will not have any effect on the audio. Assuming 32 bits sample wire format, of course.

[robiwano]

My idea as a picture :)

[flatmax]

Hi Robert,

IT is a good idea. Did you look at the sync. code I sent through ?

On 16/4/20 5:43 pm, Robert Bielik wrote:

The question is how to design the sync noise so that it doesn't
hurt. Just setting the LSB to 1 on channel one and to 0 everywhere
else might already be an improvement, but I wonder if that slight
DC offset could be avoided.

Since the CODEC itself only uses 24 bits, the low 8 bits of the I2S data will not have any effect on the audio.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Audio-Injector/Octo/issues/8#issuecomment-614473332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFLUB5FDWBOX3YXPJQF35TRM2ZKJANCNFSM4E2YVQBA.

[robiwano]

Yes. So I assume if the inputs are in need of re-alignment, the outputs should be re-aligned the same way?

[flatmax]

It sould do, but it needs to be tested and diagnosed.

On 17/4/20 3:29 pm, Robert Bielik wrote:

Yes. So I assume if the inputs are re-aligned in correct order, the output ordering will follow?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Audio-Injector/Octo/issues/8#issuecomment-615050925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFLUB2X6DUA73AIGM5WR6LRM7SNLANCNFSM4E2YVQBA.

[tiger-seo]

@flatmax hello. please, advise if this bug is gonna be fixed ? or maybe suggest workaround. thank you!

[flatmax]

Thaks for the interest. There are a few work arounds. For input audio, you can use the code from ALSAFullDuplexMinScan.C to align the input audio channels into the sequence you expect : https://github.com/flatmax/gtkiostream/blob/master/test/ALSAFullDuplexMinScan.C

Alternatively you can also try using the jackd startup method others use which keeps a persistent jackd instance running, which if it doesn't xrun will stay locked : https://github.com/Audio-Injector/Octo/issues/1#issuecomment-455135061

[flatmax]

Oh by the way, if you haven't heard, we have launched a phono preamplifier and pro. soundcard which may interest you : https://www.kickstarter.com/projects/flatmax/audio-injector-pro-sound-card-with-a-phono-analogue-skin?ref=github

[tiger-seo]

@flatmax a few minutes ago i've got channels swapped :( jackd didn't stop neither crashed, it simply started sound on different output port without any interruption. in /var/log/syslog i have following right in time when it occured:

Oct  7 18:46:24 multiroom kernel: [725278.835500] bcm2835-i2s 3f203000.i2s: I2S SYNC error!
Oct  7 18:46:24 multiroom kernel: [725278.837526] bcm2835-i2s 3f203000.i2s: I2S SYNC error!

BTW, my raspi is not loaded at all

$ uptime
 17:02:33 up 8 days,  9:44,  2 users,  load average: 0.06, 0.05, 0.02

i'm trying to restart jackd and I2S sync is happening again during restart, so channels are not in expected order

Oct  7 17:00:17 multiroom systemd[1]: Stopping LSB: JACK Audio Connection Kit...
Oct  7 17:00:17 multiroom jack[9141]: Stopping JACKD: JACK STOPPED OK
Oct  7 17:00:18 multiroom systemd[1]: jack.service: Succeeded.
Oct  7 17:00:18 multiroom systemd[1]: Stopped LSB: JACK Audio Connection Kit.
Oct  7 17:00:18 multiroom systemd[1]: Starting LSB: JACK Audio Connection Kit...
Oct  7 17:00:18 multiroom jack[9146]: Starting JACK server:.
Oct  7 17:00:19 multiroom kernel: [726112.952818] bcm2835-i2s 3f203000.i2s: I2S SYNC error!
Oct  7 17:00:19 multiroom kernel: [726112.954083] bcm2835-i2s 3f203000.i2s: I2S SYNC error!
Oct  7 17:00:19 multiroom systemd[1]: Started LSB: JACK Audio Connection Kit.

[flatmax]

I have the feeling this is to do with suspend or pause possibly due to xruns. Can you please use the "non-stop-clocks" option when you specify the overlay in your /boot/config.txt file :

dtoverlay=audioinjector-addons,non-stop-clocks

From here :

https://github.com/raspberrypi/linux/issues/2409#issuecomment-370179490

If you want to look more into the I2S SYNC error check here :

https://github.com/raspberrypi/linux/blob/rpi-5.4.y/sound/soc/bcm/bcm2835-i2s.c#L208

It is to do with DMA.

On 8/10/20 2:52 am, tiger-seo wrote:

@flatmax https://github.com/flatmax a few minutes ago i've got channels got swapped :( jackd didn't stop neither crashed, it simply started sound on different output port without any interruption. in /var/log/syslog i have following right in time when it occured:

|Oct 7 16:46:24 multiroom kernel: [725278.835500] bcm2835-i2s 3f203000.i2s: I2S SYNC error! Oct 7 16:46:24 multiroom kernel: [725278.837526] bcm2835-i2s 3f203000.i2s: I2S SYNC error! |

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Audio-Injector/Octo/issues/8#issuecomment-705028971, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFLUBZRIQWPQSFBANDDK4TSJSFD5ANCNFSM4E2YVQBA.

[tiger-seo]

@flatmax thank u, will try your suggesting

[tiger-seo]

@flatmax unfortunately , dtoverlay=audioinjector-addons,non-stop-clocks didn't fix channels swap

[nettings]

@flatmax, correct me if I'm wrong, but isn't "non-stop-clocks" meant to ensure the clocks keep running when using the ALSA device intermittently? In the jack usecase, you pretty much already get the same effect: the device is started once, and kept alive for the lifetime of jackd, so I'm not surprised it doesn't fix @tiger-seo's issue (which is the same I'm seeing).

[tiger-seo]

@flatmax can u please suggest an alternative method to interact with the soundcard input and output channels, but not having this problem with inconsistent channels ? even not jack

[Mohawi]

Hi, since a few weeks now, I have the octo running 24/7 without channel switches. I use a Pre-empt-RT patched kernel and JackD with real-time scheduling. So far it works fine for me. CPU load is a bit higher, but as long as it seems to work...

JACKD_OPTIONS="-R -P75 -dalsa -dhw:audioinjectoroc -r96000 -p512 -n3"

The processes that dock into JackD are running with real-time priorities as well...

[nettings]

@Mohawi, interesting! Did you build that kernel yourself, or is someone providing them for the RPi? Note that your jackd rt priority is higher than those of the kernel irq handlers. This means jack can theoretically starve your hardware drivers, and since it is the hardware that gets out of whack when the channels are swapped, it might be even more robust to run jack at below 50 (the default priority of the irq threads in the kernel). I use 40. Given that no other userspace RT threads exist on my system, it does not make jack any more susceptible to being starved for CPU from userland.

[Mohawi]

@nettings So far I got a pre built 4.19 one from https://www.raspberrypi.org/forums/viewtopic.php?t=250927. But since it is hard to find current ones, I thought about looking into how to build them on my own.

Thank you for the remarks about the priority. I will try to reduce the priority to 40 and see.

Actually I also found a hot candidate that, when executed, triggers the channel switches for me. apt update and if there is something to update apt upgrade.

[tiger-seo]

@flatmax hi. i noticed the same channels problem when using ALSA , in particular i'm doing following:

1. invoke speaker-test -c 8 --test=wav
2. write down speakers names
3. reboot RPI
4. run speaker test again
5. write down speakers names

as the result , speakers will have different names, for example , in my case first time i ve got: 7 - Side Right 6 - Side Left and after reboot, from the same speakers , i ve got: 3 - Rear Right 2 - Rear Left

[flatmax]

People have had success ensuring the correct channel mapping using jackd. However for direct ALSA usage a different approach needs to be take.

The background is based on input channel detection - this code will reliably remap input channels placing channel 0 in the first place. The code needs to be adapted to confirm that the same applies to the output channels - in synchrony with the input channels. One student in China has already done this with a 16 bit audio signal, but it really needs to be done for the full 32 bit signal. Once the channel remapping excercise has been completed, that code needs to be placed into an ALSA external plugin, possibly using this code as the starting point. Once the plugin exists, the .asoundrc file can be created to always use that external plugin as the default device and there will be no more channel swapping.

[abelian2202]

The channel remapping hack works on inputs. My outputs are also shifted, but not the same shift as the inputs, and not repeatably the same output shift for each input shift. This with 16 bit samples, 5.10.63-v7l+ on Pi 2B and 4B.

[Mohawi]

@abelian2202 As mentioned above somewhere I made really good experiences with running the octo in a PREEMPT_RT patched linux (I can really recommand the one from kdoren. In addition, I'm running a JACKd layer between the alsa drivers and my applications. Have a look at the example setup script I made for my specific system on my GIST (multiroom audio with 3 analog stereo sources, mopidy streams and the mirrored 3 analog outs of the inputs and one output that carries the multiroom audio . Feel free to take what you need from the script. I found obligatory is the RT linux, everything related to jack, restart_octo and the asound.conf. The rest is specific to my usecase.