Test Overwitch over PipeWire

[dagargo]

Apparently there are some issues with PipeWire.

[szszoke]

The specific change that is causing issues is somewhere in the print_init_error branch.

[szszoke]

There aren't so many commits so I can just compile them one by one and see where the issue comes up.

[dagargo]

Could you try to be more specific? I don't have PipeWire installed yet.

EDIT: Just that. Thanks!

[szszoke]

The segfault happens on the commit that handles buffer size changes: https://github.com/dagargo/overwitch/commit/43a9600c8b90bdc3a2ca4e2fda22a14bcdda3bee

As for being more specific, here is the output:

$ pw-jack overwitch -d Digitone -v                     
Device: Digitone (outputs: 12, inputs: 2)
JACK sample rate: 48000
DEBUG:jclient.c:733:(jclient_run): Using RT priority 82...
DEBUG:overbridge.c:789:(overbridge_activate): Starting j2o MIDI thread...
DEBUG:overbridge.c:797:(overbridge_activate): Starting audio and o2j MIDI thread...
[1]    341343 segmentation fault (core dumped)  pw-jack overwitch -d Digitone -

There have been some unreleased changes in PipeWire regarding buffer size callbacks.

I will try to compile the latest PipeWire master and see if it changes anything.

[szszoke]

That did it!

Looks like whatever the issue was, it was fixed in PipeWire master. I don't know which change fixed it but if I'd have to guess I'd say this one: https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/b71f47448cd5c443738e7984205e48eff41713da

[dagargo]

You're really fast! Nice that it is working with the master although I'm not sure if that commit is the culprit as in our scenario it'll have behaved likewise.

I'm leaving this issue open in case other issues arise with PipeWire.

[szszoke]

It is nice indeed! I think it is a good idea to keep this issue open until at least you make a new release and then we ask others to do some testing too.

I can test with a Digitone but I do not have other Elektron devices.

[dagargo]

I’ve just tried Overwitch over PipeWire on Debian Bullseye (stable) and kind of works. These are my findings.

top - 20:41:53 up 9 days,  2:13,  1 user,  load average: 4,04, 2,81, 1,94
Tasks: 380 total,   1 running, 377 sleeping,   2 stopped,   0 zombie
%Cpu(s):  1,0 us,  0,5 sy,  0,0 ni, 98,5 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :  16003,5 total,   6834,4 free,   3873,2 used,   5295,9 buff/cache
MiB Swap:  16319,0 total,  16319,0 free,      0,0 used.  11920,5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                           
 122552 user       9 -11 4634372 463064  98096 S  14,6   2,8   3:08.93 ArdourGUI                                                         
   1510 user      20   0 3243140 103744  69912 S   6,3   0,6  16:20.37 Xwayland                                                          
 123104 user      20   0  467236   9872   6344 S   4,0   0,1   0:15.46 overwitch                                                         
   1362 user      20   0 6077492 343692 126000 S   2,3   2,1  51:42.99 gnome-shell                                                       
   1196 user      20   0  133000  35720   5512 S   1,3   0,2   0:08.00 pipewire                                                          
     52 root      20   0       0      0      0 S   0,3   0,0   1:09.86 ksoftirqd/6

I’ve installed it by following the official Debian instructions from here.

This is the Overwitch output.

$ overwitch 
Device: Digitakt (outputs: 12, inputs: 2)
JACK sample rate: 48000
[W][000144959.100715][pipewire-jack.c:4743 jack_acquire_real_time_scheduling()] not implemented 140556855265024 20
[W][000144959.100737][pipewire-jack.c:4743 jack_acquire_real_time_scheduling()] not implemented 140556846872320 20
ERROR:utils.c:140:(set_rt_priority): Could not set real time priority
ERROR:utils.c:140:(set_rt_priority): Could not set real time priority
JACK buffer size: 64
[W][000144965.664818][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 15 wakeups
[W][000144965.681360][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 1 wakeups
[W][000144982.550302][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 6 wakeups
[W][000144982.558513][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 9 wakeups
[W][000144984.844442][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 1 wakeups
[W][000144993.058265][pipewire-jack.c:1095 cycle_run()] jack-client 0x7fd5ec006aa0: missed 5 wakeups

The use-case was monitoring the 8 individual tracks thru Ardour and send the master out to my Digitakt inputs.

As it can be seen, a lot of wakeups are missed but audio works. MIDI doesn’t.

Setting realtime priorities doesn’t work and for whatever reason I can’t see my audio interface in the Ardour routing window.

Being the first attempt I consider this a success. After all, PipeWire is not supported on Debian stable right now.

[serratedserenade]

There seems to be a recent update where pipewire-jack now considers jack2 a conflict which segfaults overwitch if I uninstall it (mostly hoping it would try to go through pipewire-jack like it did before).

So yeah, currently broken if you install pipewire-jack >= 0.3.43 (I think)

[szszoke]

There seems to be a recent update where pipewire-jack now considers jack2 a conflict which segfaults overwitch if I uninstall it (mostly hoping it would try to go through pipewire-jack like it did before).

So yeah, currently broken if you install pipewire-jack >= 0.3.43 (I think)

It's confusing what is uninstalled and what is not. Can you clarify?

I am running Manjaro and I just tested a fresh build of master.

I am running Pipewire 0.3.43 with pipewire-jack installed. I also have jack2 installed.

I start overwitch like this:

$ pw-jack overwitch -d Digitone

I am not getting any segfaults.

[serratedserenade]

Mmmm sounds like a me thing then, ignore me for now till I figure it out

[szszoke]

Something is broken again. An update installed PipeWire 0.3.45 on my system and now I get a segfault:

$ overwitch Digitone -v                                                                                                                                                                                                                                                           
Device: Digitone (outputs: 12, inputs: 2)
JACK sample rate: 48000
DEBUG:jclient.c:756:(jclient_run): Using RT priority 82...
DEBUG:overbridge.c:932:(overbridge_activate): Starting j2o MIDI thread...
DEBUG:overbridge.c:940:(overbridge_activate): Starting audio and o2j MIDI thread...
[1]    192751 segmentation fault (core dumped)  overwitch Digitone -v

I got a Digitakt in the meantime which fails the same way.

With some PipeWire logging enabled, it seems that the segfault happens after calls to jack_get_cycle_times() are made.

[szszoke]

Hail for debug symbols!

The segfault happens at jclient.c:634:

630   for (int i = 0; i < nframes; i++)
631     {
632       for (int j = 0; j < jclient->ob.device_desc->outputs; j++)
633     {
634       buffer[j][i] = *f;
635       f++;
636     }
637     }
638 

[szszoke]

f is a nullptr and dereferencing it causes the segfault.

[szszoke]

jclient->bufsize is also zero for some reason.

[szszoke]

I commented out the line at jclient.c:634 and I noticed that jclient_set_buffer_size_cb is never called on startup. This callback is the only place where jclient->bufsize is assigned so I would guess this could be the cause of the segfault.

[szszoke]

I did some more testing.

With PipeWire 0.3.43 the buffer size callback is invoked after the sample rate callback. With PipeWire 0.3.44 the segfault appears.

I will use git-bisect to find the commit that broke things.

[szszoke]

Here is the PipeWire commit that is causing the segfault: link

Here is the commit message:

jack: remember last return from jack_get_buffer_size

Remember what we last returned from jack_get_buffer_size and only
emit a buffersize change event when somwthing new is configured.

Fixes startup of jconvolver.

Fixes #1989

Now, here is a quote from the JACK API docs (source):

Tell JACK to call bufsize_callback whenever the size of the the buffer that will be passed to the process_callback is about to change. Clients that depend on knowing the buffer size must supply a bufsize_callback before activating themselves

The way I understand this is that the buffer-size callback would only be called if the buffer size is about to change and not necessarily when a JACK client is activated.

I don't have much experience with programming for JACK but maybe JACK clients are expected to query the buffer size themselves via jack_get_buffer_size if they need to know it outside of the callback functions, where it is supplied as a parameter.

If Overwitch would query the buffer size itself and not depend on the callback for setting the initial value then the cause of the segfault would be fixed. This fix would most likely not break Overwitch on top of the real JACK server either.

As a quick hack, I tried to query the buffer size right before overwitch_activate is called but I am getting a different segfault, so a bit of a refactoring is probably needed.

...
jclient->bufsize = jack_get_buffer_size(jclient->client);
jclient_reset_buffers (jclient);

if (overbridge_activate (&jclient->ob, jclient->client, jclient->priority))
...

A way to work around this issue could be to do the steps that are done in the buffer size and the sample rate callback before the JACK client is activated and then exit early from the callbacks if the same values are passed in, i.e. jclient->buffersize == nframes and jclient->samplerate == nframes.

[dagargo]

Sorry for taking so long to answer.

The way I understand this is that the buffer-size callback would only be called if the buffer size is about to change and not necessarily when a JACK client is activated.

I never thought of this interpretation.

As a quick hack, I tried to query the buffer size right before overwitch_activate is called but I am getting a different segfault, so a bit of a refactoring is probably needed.

A way to work around this issue could be to do the steps that are done in the buffer size and the sample rate callback before the JACK client is activated and then exit early from the callbacks if the same values are passed in, i.e. jclient->buffersize == nframes and jclient->samplerate == nframes.

As you wrote, it's possible to call jack_get_buffer_size once and that's all. But since Overwitch needs to know the buffer size at any time we need the callback.

A small refactor was needed but now it will work with JACK alike servers that do not call the callbacks at client activation.

Hopefully fixed in cc7c99bf880bc9cf9d5878659ca07066ca39b01f.

Check it out and let new know if it works.

[szszoke]

I will test your changes soon.

I think there is a misunderstanding about what I said.

Of course, Overwitch needs to know if the buffer size changes.

What I meant was that Overwitch could get the initial buffer size on startup, do it's initialization and then subscribe to changes.

[dagargo]

If Overwitch would query the buffer size itself and not depend on the callback for setting the initial value

Sorry for the misunderstanding. I misread that line.

[szszoke]

Your last commit fixed it!

I tested with a Digitakt first and then with a Digitakt and Digitone together.

Nice work!

[dagargo]

@tgirod has reported to work it nicely with PipeWire too in #26.

I'm closing this as verified.

[tgirod]

@tgirod has reported to work it nicely with PipeWire too in #26.

I'm closing this as verified.

Can confirm, on archlinux with pipewire 1:0.3.51 and pipewire-jack as a dropin replacement for jack. everythin appears correctly in qpwgraph too.

[aendra-rininsland]

Just writing a heartfelt thank-you to @dagargo and @szszoke for the incredibly detailed troubleshooting in this thread — my partner and I have been banging our heads against Raspbian's Pipewire implementation for days now, and we realised as a result of the excellent reporting here that it was due to the ancient version of Pipewire Raspbian ships.

Easily fixed though!

1. Add bookworm-backports sources
2. sudo apt update
3. sudo apt install pipewire/bookworm-backports libpipewire-0.3-modules/bookworm-backports pipewire-bin/bookworm-backports pipewire-jack/bookworm-backports

We now have this working splendidly on a Raspberry Pi 5! 🎉

[dagargo]

Thanks for your kind words, @aendra-rininsland.

It's always nice to read that other people find this project useful but it's even better when it works so well for them!