Occasional garbled audio on P25 trunked system

[W3AXL]

I have a trunk-recorder instance that works pretty well, most of the time. It's following a large multi-site P25 system and tracking ~25 talkgroups, and I'd say 95% of the time everything works fine.

However, I've noticed from the beginning that occasionally I'll get extremely distorted and garbled audio, and I've attached several files that show this issue (bad-decodes.zip). It almost seems like the OP25 demodulator fails to decode the voice traffic properly and instead outputs random garbled noise.

No errors appear in the trunk-recorder logs when this occurs, and it seems like all the involved processes think everything is normal.

Curious if others have experienced this issue, and what some good troubleshooting steps to take would be. It's going to be tricky to reproduce since it's relatively infrequent compared to most traffic on the system.

[tadscottsmith]

That kind of sounds like those are stuck open mics and the normalization is raising the volume way up? When that happens what does the source list look like in the JSON?

[W3AXL]

Here's the associated JSON for one of those calls:

{
"freq": 856937500,
"start_time": 1650026369,
"stop_time": 1650026378,
"emergency": 0,
"encrypted": 0,
"call_length": 9,
"talkgroup": 10106,
"talkgroup_tag": "IMPD NO1",
"talkgroup_description": "North District Primary",
"talkgroup_group_tag": "Law",
"talkgroup_group": "IMPD North",
"audio_type": "digital",
"short_name": "idps-marion",
"freqList": [{ "freq": 856937500, "time": 1650026369, "pos": 0.0, "len": 9, "error_count": 0.0, "spike_count": 0.0}],
"srcList": [ {"src": 790006, "time": 1650026369, "pos": 0, "emergency": 0, "signal_system": "", "tag": ""} ]
}

[tadscottsmith]

It really sounds like an open mic that is getting normalized much louder. Do you have a high callTimeout set in your config?

[W3AXL]

I don't have any callTimeout specified so it should be using whatever is the default value.

I've also noticed occasionally certain transmissions will be extremely quiet - even ones from the same unit. That happens much more infrequently though. Could there maybe be a larger issue with the AGC?

[tadscottsmith]

Are you using AGC? The documentation indicates it's not recommended. Can you post your config?

[W3AXL]

Sorry, I meant the audio normalization routine.

Here's my sanitized config:

{
    "ver": 2,
    "sources": [{
        "center": 856125000,
        "rate": 2560000,
        "error": -200,
        "gain": 30,
        "digitalRecorders": 8,
        "driver": "osmosdr",
        "device": "rtl=1001"
    }, {
        "center": 858825000,
        "rate": 2560000,
        "error": -200,
        "gain": 30,
        "digitalRecorders": 8,
        "driver": "osmosdr",
        "device": "rtl=1002"
    }],
    "systems": [{
        "control_channels": [857662500,856162500,856512500,858187500],
        "type": "p25",
        "talkgroupsFile": "talkgroups.csv",
        "shortName": "idps-marion",
        "modulation": "qpsk",
        "recordUnknown": false,
        "hideEncrypted": true,
        "hideUnknownTalkgroups": true,
        "compressWav": true,
        "uploadScript": "/home/w3axl/trunk-build/clean_wavs.sh",
        "digitalLevels": 1,
        "transmissionArchive": false,
        "talkgroupDisplayFormat": "id_tag",
        "audioArchive": true
    }],
    "captureDir": "/media/RomeoBravo/trunk-recorder/",
    "plugins": [{
        "name": "rdioscanner_uploader",
        "library": "librdioscanner_uploader.so",
        "server": "http://127.0.0.1:3000",
        "systems": [{
            "shortName": "idps-marion",
            "apiKey": "",
            "systemId": 101
        }]
    }]
}

[W3AXL]

Wondering if there's anything else I can try to reduce this issue. It's very jarring to be listening to the scanner and have a loud burst of noise come across.

[tadscottsmith]

You could try modifying the call_concluder to remove all normalization, but that's probably going to reduce you're overall experience. Another thing you could probably do is modify the convert_media function to accept the full call info, rather than just the filename, and then only normalize the audio if there is more than 1 transmission.

[W3AXL]

Do you know where the normalization occurs in the code? I might look into adding some kind of audio threshold check that skips normalization on silent calls.

[taclane]

Do you know where the normalization occurs in the code? I might look into adding some kind of audio threshold check that skips normalization on silent calls.

https://github.com/robotastic/trunk-recorder/blob/36dbf43c1b41e3f2a9c15e4a267841ed27417c12/trunk-recorder/call_concluder/call_concluder.cc#L35

sox %s --norm=-.01 -t wav - takes the combined wav files and normalizes it to 0.01 below dBFS before passing it on to fdkaac for audio conversion. This only occurs when compressing audio to m4a.

Alternatively, since you're using Rdio-Scanner, you can try "compressWav": false in the TR system config and enabling audio compression on the RS end as it ingests each call.

[dechilders]

I was having lots of issues with a system that I monitor producing bursts of noise exactly like the ones you attached in the zip file. I was able to get rid of those by going into trunk-recorder/lib/op25_repeater/lib/p25p1_fdma.cc and enabling the other vocoder. Link to the exact line is below. Just uncomment that vocoder, comment out the "older fullrate" vocoder and recompile. This also solved tons of other issues I was having with random popping noises in transmissions, etc. It sounds much better using that vocoder, at least for me on my computer and on the system I'm monitoring.

https://github.com/robotastic/trunk-recorder/blob/8e9e770fc3a1a99340bb255807eb5ba0da8c90a0/lib/op25_repeater/lib/p25p1_fdma.cc#L690

Patch file: othervocoder.txt

[W3AXL]

Well that did get rid of the original issue. But unfortunately, it also introduced a new, different noise to scare the pants off me while I'm scanning

new_noise.zip

Are you running with the m4a conversion turned on? I'm wondering if this is the normalization routine doing the same thing as described above.

[tadscottsmith]

That doesn't sound that different from what you posted in the original bad-decodes.zip. Are you running one of the later commits that includes the fixed error counts getting to the JSON? Can you post the JSON file for one of those calls? It still seems like it's one long transmission by one source (open mic or something) getting normalized.

[W3AXL]

Here you go. It is a single call, so perhaps it is still the normalization routine running away

{
"freq": 854987500,
"start_time": 1654558425,
"stop_time": 1654558428,
"emergency": 0,
"encrypted": 0,
"call_length": 3,
"talkgroup": 11202,
"talkgroup_tag": "MARIAN PD",
"talkgroup_description": "Marian University Police Dispatch",
"talkgroup_group_tag": "Law",
"talkgroup_group": "University Law",
"audio_type": "digital",
"short_name": "idps-marion",
"freqList": [ {"freq": 854987500, "time": 1654558425, "pos": 0.00, "len": 3.06, "error_count": "22", "spike_count": "7"} ],
"srcList": [ {"src": 4916765, "time": 1654558425, "pos": 0.00, "emergency": 0, "signal_system": "", "tag": ""} ]
}

I may look into adding some kind of minimum volume threshold to the sox normalization routine. I can't imagine I'm the only person with this issue.

[Dygear]

I've had an issue like this where my CPU was down clocking because it was overheating. Once the Trunk-Recorder instance became CPU starved from the downclocking I was getting a bunch of garbled audio. So this is reproducible for me, but maybe not in the same way as it is for you.

[tadscottsmith]

Here you go. It is a single call, so perhaps it is still the normalization routine running away

{
"freq": 854987500,
"start_time": 1654558425,
"stop_time": 1654558428,
"emergency": 0,
"encrypted": 0,
"call_length": 3,
"talkgroup": 11202,
"talkgroup_tag": "MARIAN PD",
"talkgroup_description": "Marian University Police Dispatch",
"talkgroup_group_tag": "Law",
"talkgroup_group": "University Law",
"audio_type": "digital",
"short_name": "idps-marion",
"freqList": [ {"freq": 854987500, "time": 1654558425, "pos": 0.00, "len": 3.06, "error_count": "22", "spike_count": "7"} ],
"srcList": [ {"src": 4916765, "time": 1654558425, "pos": 0.00, "emergency": 0, "signal_system": "", "tag": ""} ]
}

I may look into adding some kind of minimum volume threshold to the sox normalization routine. I can't imagine I'm the only person with this issue.

Have you tried adding the "compressWav": false in the TR system config as suggested above? That would help to determine if it's the normalization that is causing the issue.