Behringer X32: Crackling playback at 48 kHz

dstra2 commented 2 years ago

Hi,

I'm using a Behringer X32. The mixer supports two sampling rates: 44.1 and 48 kHz. Via the firewire connection (JACK audio server with firewire backend), both recording and playback at 44.1 kHz work without any problems. But at 48 kHz only recording works, playback is heavily disturbed (digital artefacts).

When the mixer is connected, it is (correctly) recognised as BEHRINGER X_UF (id:6, rev:0), GUID 00156400000082f5 at fw1.0, S400 and the following modules are loaded: snd_bebob snd_firewire_lib

The problem occurs with both tested kernels: 4.15.0-167-lowlatency (Kubuntu 18.04) 5.13.1-051301-lowlatency (Ubuntu Studio 21.04)

I would appreciate any help to solve this problem.

takaswie commented 2 years ago

Thanks for filing the issue.

I've been waiting for any report about Behringer X-UF expansion card since it uses ArchWave AG (formerly known as BridgeCo AG) DM1500 ASIC to which ALSA bebob driver is bound.

Behringer X-UF expansion card

https://www.behringer.com/product.html?modelCode=P0AQK

I think you just tested with ALSA bebob driver in stock kernel up to v5.13 in Ubuntu official repository. For some BeBoB based devices (not all of them), previous driver causes the simillar issue of playback noise. The driver got changes in v5.14 kernel in its packet streaming function to improve the issue:

Changes in ALSA firewire stack at Linux kernel v5.14 release
- https://lore.kernel.org/alsa-devel/f8c93441-2f24-48d3-b6e8-76dac7441b41@sloti41n14/

In Ubuntu distribution, the future Jammy Jellyfish (22.04) release includes the newer driver. So you can not test it instantly.

Instead, would I ask you to install baCkported driver in the repository according to instruction in README.

Cheers.

dstra2 commented 2 years ago

Thanks for the feedback. I managed to install the drivers from the Github repository and do some testing. The findings are:

Playback now basically works with both 44.1 and 48 kHz.
Unfortunately, the mixing console can now only be operated reasonably stable in 16 input/output mode (instead of 32) and with a large JACK buffer (>= 512). With a smaller JACK buffer or in 32 input/output mode, continuous xruns occur.

I suspect that the driver now generates a higher CPU/DSP load, resulting in the xruns. Is there a way to reduce the CPU/DSP load of the driver?

takaswie commented 2 years ago

Hi,

Thanks for the feedback. I managed to install the drivers from the Github repository and do some testing. The findings are:

Thanks for your test.

Playback now basically works with both 44.1 and 48 kHz.

Unfortunately, the mixing console can now only be operated reasonably stable in 16 input/output mode (instead of 32) and with a large JACK buffer (>= 512). With a smaller JACK buffer or in 32 input/output mode, continuous xruns occur.

Some users report the similar issue in default settings of jackd service. I'd like you to try below additional configurations:

Enable --sync / -S option for normal mode (synchronizing to output device explicitly) to suppress flush mode
- https://github.com/jackaudio/jack2/blob/develop/common/JackServerGlobals.cpp#L162
use 3 or more for --nperiods / -n option
- https://github.com/jackaudio/jack2/blob/develop/linux/alsa/JackAlsaDriver.cpp#L726

However, it appears that hardware vendors added any customization to BeBoB firmware even if they use the same ASIC. If the additional configurations bring you no improvements, we shall need to dive into packet streaming for your extension card.

I suspect that the driver now generates a higher CPU/DSP load, resulting in the xruns. Is there a way to reduce the CPU/DSP load of the driver?

In my opinion, users of jackd tend to judge that any XRUN comes from higher load of something including driver issue, however it is not necessarily adequate to diagnose the issue. Especially, in my understanding, implementation of jackd is not preferable ALSA application nowadays. We have experienced paradigm shift for audio data processing in this two decades, but the basic design of jackd has not been changed during the term. In this meaning, I reccomend users to migrate to PipeWire.

Anyway I'm waiting for your report about the additional two options.

As another topic, I collect the content of configuration ROM for devices connected to IEEE 1394 bus.

https://github.com/takaswie/am-config-roms/

The collection is used to generate systemd hardware database to configure udevd to make expected access permission of Linux firewire character device (/dev/fw[0-9]+):

https://github.com/systemd/systemd/blob/main/hwdb.d/80-ieee1394-unit-function.hwdb

We already have the entry corresponding to X-UF card for Behringer X32:

https://github.com/systemd/systemd/blob/main/hwdb.d/80-ieee1394-unit-function.hwdb#L162

However in case, it is better to store data of ROM for your device. You can pick it up via sysfs node, typically:

$ cat /sys/bus/firewire/devices/fw1/config_rom > behringer-x-uf.img

dstra2 commented 2 years ago

Thanks for the instructions.

use 3 or more for --nperiods / -n option

All tests (including the previous ones) were performed with 3 periods.

Enable --sync / -S option for normal mode (synchronizing to output device explicitly) to suppress flush mode

The xruns have disappeared and the JACK buffer can be set much smaller again (both in 16 and 32 input/output modes). After a while, however, noise reappears (not xruns). The time it takes for the noise to reappear is irregular (a few seconds to several minutes; I have not yet been able to see a pattern). A change in the mixer configuration, for example switching the samplerate from 48 to 44.1 kHz and back again, so that the audio backend of the Linux system is updated/reconfigured, and a subsequent restart of JACK in most cases cause the noise to disappear temporarily.

However in case, it is better to store data of ROM for your device. You can pick it up via sysfs node, typically: $ cat /sys/bus/firewire/devices/fw1/config_rom > behringer-x-uf.img

Here is the file: behringer-x-uf.tar.gz

Kind regards

dstra2 commented 2 years ago

I closed the issue by mistake and therefore reopen it...

takaswie commented 2 years ago

Thanks for the instructions.

use 3 or more for --nperiods / -n option

All tests (including the previous ones) were performed with 3 periods.

Enable --sync / -S option for normal mode (synchronizing to output device explicitly) to suppress flush mode

The xruns have disappeared and the JACK buffer can be set much smaller again (both in 16 and 32 input/output modes). After a while, however, noise reappears (not xruns). The time it takes for the noise to reappear is irregular (a few seconds to several minutes; I have not yet been able to see a pattern). A change in the mixer configuration, for example switching the samplerate from 48 to 44.1 kHz and back again, so that the audio backend of the Linux system is updated/reconfigured, and a subsequent restart of JACK in most cases cause the noise to disappear temporarily.

Hm. Would I ask you to use the other sound services like PulseAudio (or PipeWire) as comparison experiments? You can check current clock of sampling frequency by output of /proc/asound/cardX/firewire/clock (X is numeric ID for your device in Linux sound subsystem).

However in case, it is better to store data of ROM for your device. You can pick it up via sysfs node, typically: $ cat /sys/bus/firewire/devices/fw1/config_rom > behringer-x-uf.img

Here is the file: behringer-x-uf.tar.gz

Great. I pushed it at a commit c83dbf04db00.

https://github.com/takaswie/am-config-roms/commit/c83dbf04db00845180757aa1d5e33e5575492409

dstra2 commented 2 years ago

Hi,

Hm. Would I ask you to use the other sound services like PulseAudio (or PipeWire) as comparison experiments?

Tests under PulseAudio have led to the following results (playback of an audio file using Audacity, kernel 5.13.1-051301-lowlatency, drivers from this repository):

Mixer configuration	Audio file	Result
16 I/O channels, 44.1 kHz	44.1 kHz	O.k. (no noise)
16 I/O channels, 44.1 kHz	48 kHz	Playback not possible (Audacity error message due to sample rate mismatch)
16 I/O channels, 48 kHz	44.1 kHz	noise appears after a few seconds
16 I/O channels, 48 kHz	48 kHz	noise appears after about 1 minute
32 I/O channels, 44.1 kHz	44.1 kHz	O.k. (no noise)
32 I/O channels, 44.1 kHz	48 kHz	Playback not possible (Audacity error message due to sample rate mismatch)
32 I/O channels, 48 kHz	44.1 kHz	noise from the beginning
32 I/O channels, 48 kHz	48 kHz	noise continually comes and goes

Changing the mixer configuration, so that the audio backend of the Linux system is updated/reconfigured, causes the noise to disappear temporarily (as under JACK).

Kind regards

takaswie commented 2 years ago

Hi @dstra2,

Thanks for the feedback. It's possible that the device has any quirk between packet processing function and audio signal processing function since it generates noise even if packet streaming is successful without any XRUNs.

I'm preparing some tools to diagnose internal state of the device. I'd like you to wait a bit.

Thanks

takaswie commented 2 years ago

Hi @dstra2,

I'm preparing some tools to diagnose internal state of the device. I'd like you to wait a bit.

I pushed the tools into hinawa-utils repository:

https://github.com/takaswie/hinawa-utils

As a prerequisite, libhinawa 2.0 or later should be installed:

https://github.com/alsa-project/libhinawa

In Ubuntu 21.04 or later, deb pfackages (libhinawa2, gir1.2-hinawa-3.0) are available from official repository:

https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=libhinawa

Python 3 and PyGObject (python3-gi) should be required as well.

When checkout the master branch hinawa-utils repository, below scripts are available:

$ ./hinawa-bebob-plug-parser /dev/fw1 2
$ ./hinawa-bebob-connection-cli /dev/fw1 dump-connections

I'd like you to retrieve output from the above utilities, then paste them here.

Thanks

takaswie commented 2 years ago

$ ./hinawa-bebob-plug-parser /dev/fw1 2

Incidentally, the above script takes longer to finish, several seconds.

dstra2 commented 2 years ago

Hi @takaswie,

Thanks for your assistance.

When checkout the master branch hinawa-utils repository, below scripts are available:

$ ./hinawa-bebob-plug-parser /dev/fw1 2 $ ./hinawa-bebob-connection-cli /dev/fw1 dump-connections

I'd like you to retrieve output from the above utilities, then paste them here.

Here is the output of the two scripts:

Kind regards

takaswie commented 2 years ago

Thanks for the files.

According to the records of files, I can not find any issue which might cause the problem. The formation of internal plugs is really standard for BeBoB firmware without any customization by hardware vendor. Additionally I realize that the included information is quite similar to the one from FIREPOWER FCA610.

As the next step, let us investigate packet streaming itself. ALSA bebob driver supports Linux tracepoints (precisely snd-firewire-lib supports) and we can record packet data by userspace tools; e.g. trace-cmd. Would I ask you to follow below instructions?

At first, let you install trace-cmd deb package in advance. Before investigation, please ensure that no ALSA PCM application runs to use your device via ALSA bebob driver, then execute below command:

$ sudo trace-cmd record -e snd_firewire_lib:amdtp_packet

Now you can start ALSA PCM application; e.g. jackd. After a while, cancel trace-cmd process by hitting Ctrl + C.

You can see trace.dat in current working directory. Please share it. If you are interested in it, below command dump saved data:

$ trace-cmd report

Thanks

takaswie commented 2 years ago

To avoid losing the files:

output_hinawa-bebob-connection-cli.log output_hinawa-bebob-plug-parser.log

dstra2 commented 2 years ago

Hi @takaswie,

$ sudo trace-cmd record -e snd_firewire_lib:amdtp_packet

I played back a 30 second sine sweep (48kHz) using Audacity. Here is the output of trace-cmd: trace.dat.gz

Kind regards

takaswie commented 2 years ago

I played back a 30 second sine sweep (48kHz) using Audacity. Here is the output of trace-cmd: trace.dat.gz

Thank you to get the packet dump. I analyze it by the script I wrote in https://github.com/takaswie/snd-firewire-improve/issues/38#issuecomment-917610783. As a result:

ch 0:
  discontinue
  3 sec, 5076 cycle: 48000 events
  4 sec, 5076 cycle: 48000 events
  5 sec, 5076 cycle: 47784 events
  6 sec, 5076 cycle: 47928 events
  7 sec, 5076 cycle: 48000 events
  0 sec, 5076 cycle: 48000 events
  1 sec, 5076 cycle: 48000 events
  2 sec, 5076 cycle: 48000 events
  3 sec, 5076 cycle: 48000 events
  4 sec, 5076 cycle: 48000 events
  5 sec, 5076 cycle: 48000 events
  6 sec, 5076 cycle: 48000 events
  7 sec, 5076 cycle: 48000 events
  discontinue
  3 sec, 2374 cycle: 48000 events
  4 sec, 2374 cycle: 48000 events
  5 sec, 2374 cycle: 47952 events
  6 sec, 2374 cycle: 48000 events
  7 sec, 2374 cycle: 48000 events
  0 sec, 2374 cycle: 48000 events
  1 sec, 2374 cycle: 48000 events
  2 sec, 2374 cycle: 48000 events
  3 sec, 2374 cycle: 48000 events
  4 sec, 2374 cycle: 48000 events
  5 sec, 2374 cycle: 48000 events
  6 sec, 2374 cycle: 48000 events
  7 sec, 2374 cycle: 47992 events
  0 sec, 2374 cycle: 48000 events
  1 sec, 2374 cycle: 48000 events
  2 sec, 2374 cycle: 48000 events
  3 sec, 2374 cycle: 48000 events
  4 sec, 2374 cycle: 48000 events
  5 sec, 2374 cycle: 48000 events
  6 sec, 2374 cycle: 48000 events
  7 sec, 2374 cycle: 48000 events
  0 sec, 2374 cycle: 48000 events
  1 sec, 2374 cycle: 48000 events
  2 sec, 2374 cycle: 48000 events
  3 sec, 2374 cycle: 48000 events
  4 sec, 2374 cycle: 48000 events
  5 sec, 2374 cycle: 48000 events
  6 sec, 2374 cycle: 48000 events
  7 sec, 2374 cycle: 48000 events
  0 sec, 2374 cycle: 48000 events
  1 sec, 2374 cycle: 48000 events
  2 sec, 2374 cycle: 48000 events
  3 sec, 2374 cycle: 48000 events
  4 sec, 2374 cycle: 48000 events
  5 sec, 2374 cycle: 48000 events
ch 1:
  discontinue
  discontinue
  3 sec, 7032 cycle: 48000 events
  4 sec, 7032 cycle: 48000 events
  5 sec, 7032 cycle: 47736 events
  6 sec, 7032 cycle: 47976 events
  7 sec, 7032 cycle: 48000 events
  0 sec, 7032 cycle: 48000 events
  1 sec, 7032 cycle: 48000 events
  2 sec, 7032 cycle: 48000 events
  3 sec, 7032 cycle: 48000 events
  4 sec, 7032 cycle: 48000 events
  5 sec, 7032 cycle: 48000 events
  6 sec, 7032 cycle: 48000 events
  7 sec, 7032 cycle: 48000 events
  discontinue
  discontinue
  3 sec, 5011 cycle: 47992 events
  4 sec, 5011 cycle: 48000 events
  5 sec, 5011 cycle: 47952 events
  6 sec, 5011 cycle: 48000 events
  7 sec, 5011 cycle: 48000 events
  0 sec, 5011 cycle: 48000 events
  1 sec, 5011 cycle: 48000 events
  2 sec, 5011 cycle: 48000 events
  3 sec, 5011 cycle: 48000 events
  4 sec, 5011 cycle: 48000 events
  5 sec, 5011 cycle: 48000 events
  6 sec, 5011 cycle: 48000 events
  7 sec, 5011 cycle: 48000 events
  0 sec, 5011 cycle: 48000 events
  1 sec, 5011 cycle: 48000 events
  2 sec, 5011 cycle: 48000 events
  3 sec, 5011 cycle: 48000 events
  4 sec, 5011 cycle: 48000 events
  5 sec, 5011 cycle: 48000 events
  6 sec, 5011 cycle: 48000 events
  7 sec, 5011 cycle: 47992 events
  0 sec, 5011 cycle: 48000 events
  1 sec, 5011 cycle: 48000 events
  2 sec, 5011 cycle: 48000 events
  3 sec, 5011 cycle: 48000 events
  4 sec, 5011 cycle: 48000 events
  5 sec, 5011 cycle: 48000 events
  6 sec, 5011 cycle: 48000 events
  7 sec, 5011 cycle: 48000 events
  0 sec, 5011 cycle: 48000 events
  1 sec, 5011 cycle: 48000 events
  2 sec, 5011 cycle: 48000 events
  3 sec, 5011 cycle: 48000 events
  4 sec, 5011 cycle: 48000 events
  5 sec, 5011 cycle: 48000 events

Except for the discontinuities, ALSA bebob driver and x-uf exten]sion card communicates mostly with 48000 PCM frames (precisely events) per second in isochronous cycle for both directions. I can see that the noise does not come from invalid number of PCM frames over the communication.

I'm going to analyze the packet dump further. Thanks for your cooperation and patience.

takaswie commented 2 years ago

Hi @dstra2 ,

I'm under analyzing the packet dump to identify cause of the issue, however it's not cleared yet...

For my information, would I ask you to retrieve packet dump in 44.1 kHz case?

Thanks for your patience.

dstra2 commented 2 years ago

Hi @takaswie,

I played back a 30 second sine sweep again (using Audacity), this time at 44.1 kHz (both file and mixer samplerate).

You can find the output of $ sudo trace-cmd record -e snd_firewire_lib:amdtp_packet here: trace.dat(44.1kHz).gz

Please let me know if you need further dumps or if there is the need for further tests.

Kind regards

takaswie commented 2 years ago

Hi @takaswie,

I played back a 30 second sine sweep again (using Audacity), this time at 44.1 kHz (both file and mixer samplerate).

You can find the output of $ sudo trace-cmd record -e snd_firewire_lib:amdtp_packet here: trace.dat(44.1kHz).gz

Thanks for the dump file.

In both case, ALSA bebob driver communicates to the device at the same frequency set in the device. So we should find the other causes.

Please let me know if you need further dumps or if there is the need for further tests.

OK. I'd request test if I found suspicious causes. Thanks for your patience.

takaswie / linux-firewire-dkms

Behringer X32: Crackling playback at 48 kHz #42