BUG 4.1.2 RDAirPlay still sometimes chops off the end of carts during a segue

[vizubeat]

Original bug issue: https://github.com/ElvishArtisan/rivendell/issues/919

Created a new issue because I can't reopen the old one. Feel free to close/delete this issue if needs be.

I'm afraid I'm still seeing this on a fresh installation of 4.1.2 on Jammy (with imported database and audio).

Just heard it on the segue between a dry sponsor tag and music - the last 0.5s of the sponsor tag was chopped off. This meant the sponsor tag's words "musical events" became "musical ev", and it sounds very clunky.

Re-cueued the same cars three times with no issue, the glitch / early finish happened again on the fourth re-cue.

I adjusted the segue marker for the sponsor tag and re-saved, but the same problem appears.

I'm happy to send over any other information you might find useful, just let me know. But nothing seen in the logs so far.

Thanks.

Originally posted by @vizubeat in https://github.com/ElvishArtisan/rivendell/issues/919#issuecomment-1837553506

[lowlevelm]

I am also facing this issue on rocky linux 8. The issue appears to occur randomly. When setting the pie chart to count to transition it seems to synchronize with the premature transition.

[ElvishArtisan]

'Kay, so @lowlevelm is reporting this on Rocky8. @vizubeat, what is your environment?

[BigLKev]

Same issue here. Running Ubuntu 22.04 and V4.1.2 - soundcard is a Focusrite Scarlet18i20. I'm using a Dell Optiplex 3000 i5-12500T with 16gb RAM. This is also a database that was brought over from V3 that didn't have this issue. Again, it is random. System is a "standalone" install. Not using JACK but using the ALSA Scarlet Gen 2/3 Control Panel GUI from Geoffrey Bennett on Github https://github.com/geoffreybennett/alsa-scarlett-gui/blob/master/USAGE.md

[lowlevelm]

Since it seems a few of us are using converted old databases, it could be beneficial to investigate whether creating a new database might resolve the issue. I will experiment with this later this week and see if it has any effect.

I will also try hosting the audio store locally to see if it has any impact on the issue.

[ElvishArtisan]

Also relevant: the sound setup --i.e. are you using HPI, JACK or ALSA, along with the hardware specifics. I've been testing this a good bit here with HPI (mostly ASI6685 cards) and haven't heard the problem, so perhaps a driver-specific issue?

[vizubeat]

@ElvishArtisan apologies for the lack of detail - this is a fresh install of Ubuntu 22.04 Jammy, all up-to-date, ALSA with Behringer U-CONTROL UCA222 Ultra-Low Latency 2 In/2 Out USB Audio Interface on Dell Optiplex 3040 Core i5 6500 with 8GB RAM - client/server setup in this instance, but also heard on standalone install on a Dell laptop with the same Jammy/ALSA/Behringer setup. But, exactly the same hardware running Rivendell v3.6.8 on CentOS 7, no playback issues reported. Hope that helps, let me know if I can provide any other detail.

[lowlevelm]

I've tested on 2 setups:

Dell Optiplex 7050.

• Rocky linux 8
• ugreen usb audio adapter (cmedia hs100b)
• alsa
• networked

  Virtual machine

• Rocky linux 8
• jack (Rivendell>stereotool>glasscoder) (dummy audio device)
• networked

I am running a database and audio store both in separate lxc containers on proxmox. Proxmox is running on a dell poweredge r410 rack server, hosting the containers, as well as my production vm running rivendell 3.

I have a video of the glitch occurring, and upon inspection the pie chart does not sync to the transition, so my initial comment was mistaken. The transition between the song "Ready Set No" and the liner starts about 0.6s too early, and the transition to the next song is just fine. The bug only occurs about 1 in every 3 transitions.

[ElvishArtisan]

Ok, I have a test that I'd like those who are experiencing this problem on v4.1.2 to run:

1) Close all Rivendell windows.

2) Open '/etc/rd.conf', find the [Hacks] section, and add the following line to it: PadSegueOverlaps=500 Save the change.

3) Restart the Rivendell service by doing (as 'root'): systemctl restart rivendell

4) Start rdairplay and test. Are you still hearing the problem? If, has it changed in severity?

[BigLKev]

Ok, I have a test that I'd like those who are experiencing this problem on v4.1.2 to run:

1. Close all Rivendell windows.
2. Open '/etc/rd.conf', find the [Hacks] section, and add the following line to it: PadSegueOverlaps=500 Save the change.
3. Restart the Rivendell service by doing (as 'root'): systemctl restart rivendell
4. Start rdairplay and test. Are you still hearing the problem? If, has it changed in severity?

Hi Fred, I've done as you suggested. It does seem to have improved on my system. The cuts that were affected were "PLAY" transitions with news, adverts or downloaded spoken links. So far, none have been chopped short since making the change. I will keep monitoring. Thank you.

[vizubeat]

Thanks @ElvishArtisan this is much improved, but still not quite right.

Without PadSegueOverlaps=500 I had plays 1,3,5,8,10 out of 10 plays sound 'chopped' - there was very clearly something missing from the end of cart 1 into cart 2 at the point of the transition.

With PadSegueOverlaps=500 set, all 10 plays sounded much better, but on a handful of plays there was still something missing from the end of cart 1 into cart 2 at the point of the transition.

So I recorded the output of Rivendell to try and visualise this better. Hopefully the images with labels explain the situation.

1:

2:

TLDR, PadSegueOverlaps=500 sounds much improved, and probably usable, but still not quite right.

Let me know if you need any other information.

[ElvishArtisan]

Thanks for checking everyone. This does narrow things down a bit.

[lowlevelm]

Same here, I set PadSegueOverlaps to 600 and the songs I had issues with now segue properly. However, this workaround introduces a 600ms gap between tracks that segue properly.

[ElvishArtisan]

Yup. There's a reason that that setting is in the [Hacks] section. It's not a cure, more a "diagnostic aid".

[rugger]

Just figured I would add that I am also seeing this with Rocky 8, ASI6585 and 6685 cards. Haven't added the "Hack"

[rugger]

Had a cart this morning skip the segue and end marker and play the entire cart. Looking at the cart to see if I had place the end marker wrong, this is what I found. If you notice the cart has the same start/End and Length. This is a converted database from 3

[ElvishArtisan]

If you notice the cart has the same start/End and Length. This is a converted database from 3

Yet the actual marker is in the expected location on the waveform; which makes me think we've got a display issue in that dialog. That's certainly wrong, but I doubt it's factor in this particular issue.

[ElvishArtisan]

Fixed in af8b711. I think. (Hope?)

Please give this one a Real Good Test. We've done a bunch of refactoring of things in caed(8), so quite possible there's some collaterial damage to mop up from that.

[lowlevelm]

I’ve done some initial testing, and this patch seems to have corrected the issue where the end of a cut will be cut off, but I still encounter segues being fired too early/late. It does sound good enough for most stations, so long as their format doesn’t require precise timing on transitions. I plan on further testing during the weekend to see if I can narrow it down

Message ID: @.***>

[vizubeat]

I'd love to test but not sure how to. Do I need to compile from source? Are there any instructions for those of us that might not have done that before? I've only updated via yum/apt before which I guess needs updated Paravel packages.

[lowlevelm]

There are instructions here: https://wiki.rivendellaudio.org/index.php/Building_Rivendell_from_source

It’s not too hard, so long as you follow the instructions and are comfortable with using the terminal. make sure to use a new install, and one that is dedicated to testing

Message ID: @.***>

[vizubeat]

Thanks @lowlevelm - followed those instructions and I have built from source for the first time.

I'm definitely on the v4 branch as the window-too-wide problem that I flagged has been fixed (thanks @ElvishArtisan!)

But I'm afraid the chopped audio problem still exists. I don't know how else to represent the problem, but below is a screenshot of the output of Rivendell, showing the same transition between two carts - the top waveform is good, the bottom is bad, there's about 300ms missing from cart1 as it transitions into cart2 in the 'bad' transition.

Edit: PadSegueOverlaps=500 has been removed from /etc/rd.conf

[ElvishArtisan]

Thanks @lowlevelm and @vizubeat for the feedback! And, as @vizubeat notes, it is important to remove the PadSegueOverlaps= entry from /etc/rd.conf before testing.

[vizubeat]

Happy New Year everyone. @ElvishArtisan just wondering if there was any update to this? I have a set of new hardware waiting to be deployed with v4 but I don't think we're able to use it in its current state. I also contacted Paravel about a paid support plan, but haven't heard anything back yet. I'd love to get this live! Thank you.

[vizubeat]

I have a fresh install of v4.1.3 with an import of our original database from v1.7.2 and a handful of carts/cuts for testing.

I'm afraid the issue still persists: the occasional 'early finish' of cart A, on a segue from cart A to cart B.

I have tried many segues between old/old carts, old/new carts, new/old carts and new/new carts, and the issue still happens sometimes, so it doesn't feel like it's an issue with audio or legacy database values.

However, a new finding, the 'early finish' of cart A, on a segue from cart A to cart B, does not appear when there is no segue marker set on a cut in cart A. The prevalence of the 'early finish' also appears to be diminished when 'no fade on segue' is selected on a cut in cart A.

Setting either 'no segue markers' or 'no fade on segue' may well be an appropriate workaround for my use case, as the issue is most noticeable on the segue between sponsor idents and music carts, so no segue whatsoever sounds okay. But I wonder if this will be a problem on other segues from general station idents into music where a segue really is beneficial for a smooth transition.

I will continue to test with more segues between carts, but I would love to get this fixed. If there's anything else I can do to test or provide diagnostic information, do please let me know.

[ctecamp]

I am having the same issue, cuts with segue markers truncate at the marker or shortly after the marker. Rivendell 4.1.3 Ubuntu LTS Jack Audio

[lbracci]

I'm having the same issue, affecting some cuts with segue markers and segue transition. It truncates at about half of second before the end.

We are using Rivendell 4.1.3, Linuxmint 21 (based on Ubuntu 22.04 LTS), JACK. System running at 44,100Hz. Our card is a MAudio Delta 1010 but I can also reproduce the issue in a computer with internal Intel audio.

To reproduce the issue, I made a test cart of ~16 seconds having segue markers in the last 0.3 seconds; then, I made a playlist in RDAirPlay repeating the test cart 5 or 6 times. RDAirPlay plays the first cart normally, and it fades into the second cart as expected, but starting the second cart, it truncates at about half second before the segue.

When I remove the segue markers in the test cart using RDLibrary, and I play the playlist again, it doesn't truncates anymore.

EDIT: Cuts truncate also in ALSA mode. It also truncates with play transitions, when the cut has no segue markers.

[vizubeat]

Hi @ElvishArtisan just wondering if there's any update to this issue? With the problem still present in 4.1.3 we're not able to upgrade to 4.x in production. I'm also reluctant to upgrade to 3.x with the upcoming EOL of CentOS 7. Still happy to undertake some testing if that would be helpful! Many thanks.

[bpm1992]

We had to pull back out production instances to 3.4.1 because of this.
We'll keep on testing on 4.3.x and the latest build - and happy to help.

[vizubeat]

This is still present in 4.2.1 - @ElvishArtisan are you able to give us any kind of update to this, even if it's a 'will not fix'?

I have a workaround for short liners and IDs - add a marker with PLAY transition after the audio to ensure no 'chop', but this isn't an option for music segues because then there is no segue. It doesn't sound good, and again it is likely to delay us putting 4.x into production :-(

I am also still happy to pay for support from Paravel on this issue, but I haven't had a reply to my message and I don't know how else to get in touch.

[ElvishArtisan]

Further progress on this issue is going to require a fairly significant refactoring of caed(8), as we're up against some fundamental latencies imposed by Qt5's networking system. I'm planning to do that, but am currently trying clear off the backlog of existing "simple" (heh, heh) bug reports.

[vizubeat]

Thanks for the update @ElvishArtisan that's much appreciated, I'll sit tight.

Happy to be involved in testing when the time is right :-)