Problem with audio getting delayed over time? Is there a fix?

[bnelson333]

Does anybody else have this problem? I'd really like to be able to find a solve for it if anybody knows. Running sdrtrunk on Xubuntu 18, using the internal streaming setup to icecast2 server. When it first boots up, everything is great, the delay is maybe 30 seconds from real-time to hearing it over the stream, totally acceptable.

But over the course of the day, the delay gets longer and longer. By comparing the timestamps dispatch gives in their calls to the clock, by the end of the day my stream is easily 10 minutes behind. Restarting icecast2 does nothing, but restarting sdrtrunk instantly fixes it and i'm back to real time. So there seems to be something in the sdrtrunk to icecast2 send that's slowly delaying over time? Ideas?

EDIT - it actually looks like restarting icecast2 (without restarting sdrtrunk) DOES fix the problem so now i'm even more confused. Unfortunately restarting it (e.g. like on an hourly cron job) won't work because it stops the feed. I need to find another way.

[DSheirer]

When you look at the streaming status panel, do you see the queued audio count increasing? If so, you may be trying to stream too much call content and the streaming can't accept the audio fast enough. There's a max 'staleness' value that kicks in and starts to throw away audio that sets in the queue too long.

[bnelson333]

I can't look at the moment, but wouldn't that be the opposite of my problem? That is, if I was trying to push too much, shouldn't it be dropping calls in order to keep me "current"? It's almost like it's not doing that. But I don't think I'm pushing too much, there are long periods of silence in my feed because I'm only following a couple aliases on this feed. Could that have anything to do with it? Like almost not enough audio to keep the feed happy?

At the moment I disabled the burst on connect in the icecast config and that seems to have helped, tentatively. I won't really know until a whole day goes by.

[stancel]

Yes, I do the exact same thing and see this same issue on version 0.4.0 running Ubuntu 20.04 and 64 GB of RAM. Have an older Ice Lake Intel processor in this machine and Hack RF Portapack with the Mayhem firmware for my SDR.

Wish I could get it to process the threads using the NVIDIA graphics card, because at least for me; the root cause seems to be high CPU from processing a lot of signals that go through the Site ID (closest P25 repeater tower). I have looked at numerous metrics over many times of this happening and the WAIT times from CPU blocking and unblocking of threads increases dramatically over time. That is when I start seeing large delays in the audio coming across my Icecast2 stream. Like you stated in the issue, if I periodically shut down SDRTrunk after a few hours and restart it, or reboot the computer then the Blocking WAIT time (monitored with htop) are normal and there is not a large delay in the audio being sent over the Icecast2 stream. Hope that helps provide some additional information for the issue.

[bnelson333]

I was wondering if getting something a little newer / more horsepower CPU wise would help but you're already using something significantly better than what I do. If you're still having problems then it seems like it must be something else.

This problem doesn't happen when I pull the audio from the machine and feed it to icecast via darkice. It only happens when I use the internal icecast2 configuration/streaming part of sdrtrunk... If I could use darkice I would, but I want to utilize the delay function within sdrtrunk. I want my feed to always be five minutes behind.

The behavior I'm describing here though is completely separate from the delay. It does this no matter if I'm using a delay or not.

[bnelson333]

Not really sure if this is a fix or a bug or whatever, but thought I'd report back on what I found in the last day or so. I went ahead and did apt-get update, apt-get upgrade, got everything up to snuff. Then deleted sdrtrunk and installed the latest version (beta 6). I started completely over, set up all my aliases, channel, and streaming from scratch. Now here's where I think there may be something pertinent, rather than launch the executable via a .sh file, I just went into terminal, to the bin directory, and did ./sdr-trunk. Everything seems to be fine now, it's been running since yesterday on two different machines and it's at most 1 minute behind. I haven't had to restart it since.

In the past, I was launching it from a .sh file that looked like this (run from the sudo crontab):

sudo killall java
sleep 1
export _JAVA_OPTIONS="-Xms512m -Xmx4096m"
cd /home/user/sdrtrunk/bin
sudo -u user ./sdr-trunk &

The quirk about that is when you do that, it puts the whole shebang into the background. Java and sdrtrunk are running, you just can't see them or interact with them. Perhaps that puts it in a memory limited mode or something? I'm not sure why. But launching it physically from the bin folder with ./sdr-trunk seems to have done the trick (or reinstalling everything, not really sure).

[doug-hoffman]

Your issue is most likely fixed by #1244. The current behavior results in too much silence being sent, which causes the stream to get further and further behind the longer you listen.

[DSheirer]

Merged @doug-hoffman PR to resolve this issue. Closing.