search

thewierdnut / asha_pipewire_sink

Asha audio protocol implementation for linux.
The Unlicense
28 stars 3 forks source link

How I'm supposed to connect both sides? (MED-EL Hearing Device) #3

Closed haecker-felix closed 5 months ago

haecker-felix commented 5 months ago

First of all: I'm able to stream audio. It works. I'm amazed :heart: - The catch: I'm only able to get one side working :/

The problem is with connecting the hearing aids itself, not the actual streaming (means your code). I don't manage to get both sides connected, just only one side.

For testing purposes I factory reset both the devices. Now I have both sides connected, but as individual devices, means I'm not able to stream to both simultaneously.

image

I guess this is because of the factory reset, and they lost the "pair" information. Means I'd need to connect them to each other again using the mobile vendor app. But then I'd have the problem again that I'm only able to connect one side...

I wonder if I'm doing something wrong. Any pointers? Does it work for other vendors?

haecker-felix commented 5 months ago

Oh oh! I got it working for both sides!

But then I'd have the problem again that I'm only able to connect one side...

Actually, this wasn't the case!

So my connections problems were solved by this workaround:

image

Observation: When it works, the devices are getting displayed with the personalized name (in this case "Felix's Hearing Dev.") instead of the generic vendor name ("MED-EL Hearing Device")

So idk what happened here, feel free to close the issue. No idea how to debug this mess, but feel free to add this workaround to the README or wiki, maybe it helps others as well.

In either case, you can add "MED-EL AudioStream" to the wiki. It works almost perfect (after spending hours on getting them paired/connected with the computer). From time to time it stutters a bit, then it's fine again for few seconds. But no total dropouts / disconnects. (Adapter: "Intel Corporation Wi-Fi 6 AX200 (rev 1a)")

https://github.com/thewierdnut/asha_pipewire_sink/wiki

haecker-felix commented 5 months ago

Hm... Now after testing it a while, I have the feeling the mini stutters are getting worse. They also happen with only one side being connected, so I guess it's unrelated of 2M PHY. I have the feeling it's better again when you turn off bluetooth and reconnect the devices, but not sure if this is actually the case or just coincidence (or imagination).

Another random observation: Sometimes the devices are falling back to the generic vendor name when they're disconnected, but switch back to the personalized name once you connect again:

image

haecker-felix commented 5 months ago

Oh no, the the joy was short-lived. After trying to adjust Min/MaxConnectionInterval the devices instantly disconnect again after connecting, even after changing it back to the suggested 16 :(

haecker-felix commented 5 months ago

Even after rebooting I'm unable to connect to the hearing devices again. Dear god, why is all this so fragile... Somehow I have the feeling I need to do the annoying/cumbersome factory-reset dance again to get them working again.

thewierdnut commented 5 months ago

For testing purposes I factory reset both the devices. Now I have both sides connected, but as individual devices, means I'm not able to stream to both simultaneously.

This means the devices have separate sync ids. Somehow, they have to be marked as a pair, which I presume your mobile app can do.

If you connect to the devices, and then run asha_connection_test, it will dump the name, alias, and HiSync ids associated with your devices. It will also warn you if any of your settings appear to be incorrect.

thewierdnut commented 5 months ago

Even after rebooting I'm unable to connect to the hearing devices again. Dear god, why is all this so fragile... Somehow I have the feeling I need to do the annoying/cumbersome factory-reset dance again to get them working again.

Note that after rebooting, you have to set the 2M PHY every time. I'm hoping this is fixed in a newer version of bluez.

thewierdnut commented 5 months ago

Oh no, the the joy was short-lived. After trying to adjust Min/MaxConnectionInterval the devices instantly disconnect again after connecting, even after changing it back to the suggested 16 :(

I have my suspicions... Do you have audio streaming when you connect to the hearing devices? Or are you connecting, and then starting sound playback later?

thewierdnut commented 5 months ago

Hm... Now after testing it a while, I have the feeling the mini stutters are getting worse. They also happen with only one side being connected, so I guess it's unrelated of 2M PHY.

I'm having trouble getting pipewire to deliver audio at the required cadence. I'm still playing with the code, but I suspect that it occasionally falls behind and drops a packet.

If you pause playback for a second, and then start it again, does it clear up the stutters?

thewierdnut commented 5 months ago

Dear god, why is all this so fragile...

Because its brand new software, and I only have my own pair of hearing-aids to test it with. Hopefully with your help, I can make this more stable.

haecker-felix commented 5 months ago

Note that after rebooting, you have to set the 2M PHY every time. I'm hoping this is fixed in a newer version of bluez.

I didn't had to modify it, it was already set here.

I have my suspicions... Do you have audio streaming when you connect to the hearing devices? Or are you connecting, and then starting sound playback later?

No audio streaming, and your app isn't running at all. Streaming works finely from phone. I turned Bluetooth off to ensure they're not connected with the phone - but I'm still unable to get them connected with the computer again :(

If you pause playback for a second, and then start it again, does it clear up the stutters?

I'm going to test, once I get them connected with my computer again...

Because its brand new software, and I only have my own pair of hearing-aids to test it with. Hopefully with your help, I can make this more stable.

To clear up any possible misunderstanding here - I did not mean your software, more Bluetooth and my hearing aids in general ;)

haecker-felix commented 5 months ago

So... I factory reset them again and applied the above mentioned "workaround", stuff works again - at least for now. 

Factory reset hearing aids
Connect each side with computer (-> at this point no bimodal audio yet)
Use mobile vendor app to set-up both sides as one "pair" again
🪄 now bimodal streaming from computer works

If you pause playback for a second, and then start it again, does it clear up the stutters?

I don't have the feeling it makes a difference. The output gets spammed with these: 

** (process:5698): WARNING **: 22:04:33.536: Dropping frame for Felix's Hearing Dev. (Left)

** (process:5698): WARNING **: 22:04:33.579: Dropping frame for Felix's Hearing Dev. (Left)

** (process:5698): WARNING **: 22:04:33.622: Dropping frame for Felix's Hearing Dev. (Left)

** (process:5698): WARNING **: 22:04:33.792: Dropping frame for Felix's Hearing Dev. (Left)

** (process:5698): WARNING **: 22:04:33.835: Dropping frame for Felix's Hearing Dev. (Left)

** (process:5698): WARNING **: 22:04:33.963: Dropping frame for Felix's Hearing Dev. (Left)

The "dropping frame" output matches to the stuttering which I'm getting.

thewierdnut commented 5 months ago

Can I get a btmon capture? I would like to analyze the packet timing.

thewierdnut commented 5 months ago

Also, next time you have trouble getting it to stream, can you run asha_connection_test and send me the output of that too?

thewierdnut commented 5 months ago

When you get a chance, can you get the latest version and try out the new asha_stream_test utility. It has a --algorithm argument to try out different timing algorithms to try and smooth out the audio, and I am interested in which one works best for you.

haecker-felix commented 5 months ago

Just gave it a try, unfortunately all algorithms end up in a not-so-nice noise / buzzing. So I guess something else broke in meanwhile :/

haecker-felix commented 5 months ago

oh oh oh, whops, my bad, I should read the entire text. 

<raw g722 file>

I tested it with a mp3.

thewierdnut commented 5 months ago

Yeah.... it doesn't validate or decode the data, it just dumps it raw into your devices.

haecker-felix commented 5 months ago

I'm not able to notice a difference between deadline and fixed, poll sounds completely off / broken, sounds like playing the test file with 300% speed. Will do more testing once I have a bit more time available!

thewierdnut commented 5 months ago

This is why I wanted to see the results on other devices. All three methods work fine for me.

Are you having any problems getting both devices up and running that that test app? Or does it periodically disconnect them on startup?

haecker-felix commented 5 months ago

Also, next time you have trouble getting it to stream, can you run asha_connection_test and send me the output of that too?

I'm having problems connecting them to the computer again :/ 

$ ./asha_connection_test 
(process:283981): GLib-GIO-DEBUG: 19:32:40.876: Using cross-namespace EXTERNAL authentication (this will deadlock if server is GDBus < 2.73.3)

(no more output, that's it)

Note: This is about the Bluetooth connection itself, your asha_pipewire_sink isn't involved yet.

haecker-felix commented 5 months ago

I did a full factory reset (one more time...), now it works again.

I get warnings about 2M PHY, despite having it enabled

** (process:5669): WARNING **: 19:52:25.851: 2M PHY not enabled

** (process:5669): WARNING **: 19:52:26.146: 2M PHY not enabled

Both LE2MTX and LE2MRX are listed under Selected phys, and I reconnected the hearing aids. 

haeckerfelix@fedora:~/Documents/Projekte/asha_pipewire_sink/build$ sudo btmgmt phy
Supported phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LE2MTX LE2MRX LECODEDTX LECODEDRX 
Configurable phys: BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE2MTX LE2MRX LECODEDTX LECODEDRX 
Selected phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LE2MTX LE2MRX
thewierdnut commented 5 months ago

I get warnings about 2M PHY, despite having it enabled

Running btmgmt phy sets the default negotiated phys. That just means that your adapter supports it. That warning comes from checking the socket state after it starts up. It may mean nothing, as the 2M PHY status is negotiated asynchronously with the hearing device after the socket connects, and we may just be checking the status before the negotiation is complete. If it happens every time though, it may be that your hearing device does not support 2M PHY.

haecker-felix commented 5 months ago

it may be that your hearing device does not support 2M PHY.

seems so then, because I get those messages every time.

Can I get a btmon capture? I would like to analyze the packet timing.

hcitrace.zip

...
** (process:9760): WARNING **: 21:10:24.040: Dropping frame for Felix's Hearing Dev. (Right)

** (process:9760): WARNING **: 21:10:24.082: Dropping frame for Felix's Hearing Dev. (Right)

** (process:9760): WARNING **: 21:10:24.295: Dropping frame for Felix's Hearing Dev. (Right)

** (process:9760): WARNING **: 21:10:24.381: Dropping frame for Felix's Hearing Dev. (Right)

** (process:9760): WARNING **: 21:10:24.423: Dropping frame for Felix's Hearing Dev. (Right)
...
thewierdnut commented 5 months ago

Can I get a btmon capture? I would like to analyze the packet timing.

hcitrace.zip

This appears to be a capture in the middle of stream, so it does not have the connection setup negotiation that would include the parameters. If I could get a new capture that includes that connection setup, that would be great.

I do see two streams that appear to be audio streams. In general, each packet represents 20ms of of audio, so you want your average to be well below that. Both streams have packets going up into the 40ms range. If too many of those occur in a row, you will start dropping audio frames. 

        TX packets: 250/248
        TX Latency: 2-60 msec (~19 msec)
        TX size: 167-167 octets (~167 octets)
        TX speed: ~56 Kb/s

  100 +-+------------------------------------------------------------------+   
       +|                                                                  |   
       +|                                                       TX +-----+ |   
       +|                                                                  |   
       +|                                                                  |   
       +|                                                                  |   
        |                                                                  |   
        |  +                 + +                    ++                     |   
   10 +-|  |            +    | |                +   ||                     |   
       +|  |   ++  +    |+  +| |    + +      +++|+  ||                     |   
       +|  |   || +|+   ||+ ||+|  + | |   +++|||||  ||                     |   
       +|  |   ||+|||   ||| ||||  |+| |+  ||||||||  ||                     |   
       +|  |  +||||||+ +|||+||||  ||| ||  ||||||||  ||                     |   
       +|  |  |||||||| |||||||||  ||| ||+ ||||||||+ ||            +        |   
        |  |  |||||||| |||||||||  ||| ||| ||||||||| ||            |        |   
        |  |  |||||||| |||||||||  ||| ||| ||||||||| ||            |        |   
    1 +-| +|+ |||||||| ||||||||| +|||+||| ||||||||| ||          ++|+ +    +|   
       +| ||| |||||||| ||||||||| |||||||| ||||||||| ||          |||| |    ||   
       ++------------------------------------------------------------------+   
        +          +          +           +          +          +          +   
        0          10         20          30         40         50         60  
                                   Latency (ms)
        TX packets: 308/307
        TX Latency: 3-41 msec (~7 msec)
        TX size: 167-167 octets (~167 octets)
        TX speed: ~107 Kb/s

  100 +-+------------------------------------------------------------------+   
       +|                                                                  |   
       +|      +                                                TX +-----+ |   
       +|      |                    +                                      |   
       +|      | +                  |                                      |   
       +|      | |+                 |                                      |   
        |      | ||     +           |                                      |   
        |     +| || +   |    ++    +| +                                    |   
   10 +-|     || || |+  |    ||    || |                                    |   
       +|     || || || +|  + || ++ || |                                    |   
       +|   + || || || || +| || || || |                                    |   
       +|   | || || || || || || || || |+                                   |   
       +|   | || || || || || || || || || +     +                           |   
       +|   | || || || || || || || || || |+    |                           |   
        |   | || || || || || || || || || ||    |                           |   
        |   | || || || || || || || || || ||    |                           |   
    1 +-|   | || || || || || || || || || || +  |              +     ++     |   
       +|   | || || || || || || || || || || |  |              |     ||     |   
       ++------------------------------------------------------------------+   
        +      +       +      +       +      +       +      +       +      +   
        0      5       10     15      20     25      30     35      40     45  
                                   Latency (ms)
haecker-felix commented 5 months ago

If I could get a new capture that includes that connection setup, that would be great.

here you go: hcitrace.zip

Both streams have packets going up into the 40ms range.

is that something which we can influence? (I have no idea how all this works, it's basically black magic for me)

haecker-felix commented 5 months ago

I have just compared the algorithms again by playing a test file for a fixed time (on both sides). I have recorded how many frames are dropped. The number of dropped frames also matched my subjective impression of the audio quality.

  1. fixed: Feels the best. -> ~100 frames dropped
  2. deadline: Close to fixed, but has more acoustic stuttering. -> ~500 frames dropped
  3. poll: Very random. Sometimes plays at 300% speed. Definitely the worst algorithm (at least for me). -> ~20k frames dropped
thewierdnut commented 5 months ago

Both streams have packets going up into the 40ms range.

is that something which we can influence? (I have no idea how all this works, it's basically black magic for me)

I hope so. Bluetooth negotiates packets on a fixed connection interval, which should be 20ms. Pipewire however does not deliver 20ms of audio at a timer, so it periodically goes badly out of phase with the delivery schedule expected by the hearing devices. For me, it sounds great for about ten seconds, then stutters a bit, then sounds great for another ten seconds, and so on.

The "fixed" algorithm is an attempt to deliver the traffic at the expected 20ms period, which should work better. In theory, the bluetooth adapter is supposed to be managing all of this for us, but :man_shrugging:.

thewierdnut commented 5 months ago

If I could get a new capture that includes that connection setup, that would be great.

here you go: hcitrace.zip

That doesn't have it either. Can you do this:

  1. Disconnect your devices.
  2. Run btmon -w setup.snoop
  3. Attach one device (don't bother streaming audio. just the initial setup is fine)
  4. Kill btmon.
haecker-felix commented 5 months ago

Okay, trying one more time...

setup.zip

thewierdnut commented 5 months ago

Okay, trying one more time...

setup.zip

So in packets 41 and 216, I see the remote devices advertising 2M PHY support, and I know the adapter chipset supports it (I have the same chipset available to me), but I don't see bluez requesting it.

What linux distribution and kernel are you running? What are your bluez and glib version? 

uname -a
pkg-config --modversion glib-2.0
pkg-config --modversion bluez
haecker-felix commented 5 months ago

Fedora Silverblue 40

Linux fedora 6.8.11-300.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Mon May 27 14:53:33 UTC 2024 x86_64 GNU/Linux
glib2-2.80.2-1.fc40.x86_64
bluez-5.76-1.fc40.x86_64
thewierdnut commented 5 months ago

hm... all newer than mine.

thewierdnut commented 5 months ago

Feel free to test the latest code if you get a chance. I've made several changes to the queuing algorithm that I'm hoping will make it much more stable.

thewierdnut commented 5 months ago

I've added your device to the wiki. Please continue to test and let me know if you run across any other issues.