New QRA-Skimmer Multi Digital Mode Decoder/Reporter Details

Phil This isn't an issue but just somewhere in your domain that I can provide some details and start a conversation about my QRA-Skimmer app (restating I'm build on the backs of others) to hopefully help provide some feedback that might help to tune/tweak KA9Q-Radio and your supporting apps for external app usage.

In my radiod.conf file I currently set up many individual multicast addresses each for a single frequency representing a band/mode. Is there any performance hit for doing this vs an all in one as per your examples? For example I do something like, abbreviated to pertinent variables below:

[FT4_20m] data = ft4_20m.local freq = 14m080000

I have a config file in my app that defines a time span and then lists a number of band/modes to simultaneously capture/decode during that time. During the appropriate time I spawn/queue your "pcmrecord" for the interval associated with a mode along with the mode's multicast name from a template/lookup table ( [MODE_BANDm] / mode_bandm.local ) as seen above. I scan stdout and extract your generated filename.

After a record completes I then rename the file it, as required by the decoders, and pass it to the appropriate command line decoder for the associated mode. Decodes are parsed by WSJTX jt9, wsprd, or JS8CALL js8 command line apps for Linux that ingest WAV files. Their text output is processed and uploaded to the appropriate online reporting services. I also have a DXcluster like telnet server and WSJTX multicast/uni-cast broadcaster that can also take decodes and make data available through those apps/modules/protocols.

I know you have a "control" program that may be useful for changing the frequency of a multicast stream, but I could not get that to work when playing with "monitor". Therefore I'm pursuing how best to change band/mode combos and teardown old and setup new multicast streams for CPU's that might not be able to handle numerous simultaneous recorders/decoders. Currently I'm running 28 virtual band/mode receivers (160-10m)/(FT8, FT4, WSPR) on a 8-core Xeon and still have plenty of head room to add alternate/secondary band/modes above as well as new band/modes JT9, JS8, FST4W, etc, I realize not everyone will have that luxury of monitoring all band/modes all the time. So...

I am thinking about simply and programmatically generating a new radiod.conf file (using some template) to generate single stream name and single frequency sections as above and then "systemctl restart radiod" service. There does seem to be some start up time/delay in doing this, but it seems "pcmrecord" doesn't care or at least times out and eventually retries and picks up at the next available record/decode cycle. BTW, I'm using /dev/shm for all the WAV file writing as not to wear out my SSD.

So there it is in a nutshell, not rock science, but surely a fun science project :) I'll share more as requested and as I make more progress.

andyz - K1RA

I'm using

Andy...

Suggest you look at using the tune binary

-Jim

On Dec 9, 2023, 15:05, at 15:05, Andy Zwirko @.***> wrote:

Phil This isn't an issue but just somewhere in your domain that I can provide some details and start a conversation about my QRA-Skimmer app (restating I'm build on the backs of others) to hopefully help provide some feedback that might help to tune/tweak KA9Q-Radio and your supporting apps for external app usage.

In my radiod.conf file I currently set up many individual multicast addresses each for a single frequency representing a band/mode. Is there any performance hit for doing this vs an all in one as per your examples? For example I do something like, abbreviated to pertinent variables below:

[FT4_20m] data = ft4_20m.local freq = 14m080000

I have a config file in my app that defines a time span and then lists a number of band/modes to simultaneously capture/decode during that time. During the appropriate time I spawn/queue your "pcmrecord" for the interval associated with a mode along with the mode's multicast name from a template/lookup table ( [MODE_BANDm] / mode_bandm.local ) as seen above. I scan stdout and extract your generated filename.

After a record completes I then rename the file it, as required by the decoders, and pass it to the appropriate command line decoder for the associated mode. Decodes are parsed by WSJTX jt9, wsprd, or JS8CALL js8 command line apps for Linux that ingest WAV files. Their text output is processed and uploaded to the appropriate online reporting services. I also have a DXcluster like telnet server and WSJTX multicast/uni-cast broadcaster that can also take decodes and make data available through those apps/modules/protocols.

I know you have a "control" program that may be useful for changing the frequency of a multicast stream, but I could not get that to work when playing with "monitor". Therefore I'm pursuing how best to change band/mode combos and teardown old and setup new multicast streams for CPU's that might not be able to handle numerous simultaneous recorders/decoders. Currently I'm running 28 virtual band/mode receivers (160-10m)/(FT8, FT4, WSPR) on a 8-core Xeon and still have plenty of head room to add alternate/secondary band/modes above as well as new band/modes JT9, JS8, FST4W, etc, I realize not everyone will have that luxury of monitoring all band/modes all the time. So...

I am thinking about simply and programmatically generating a new radiod.conf file (using some template) to generate single stream name and single frequency sections as above and then "systemctl restart radiod" service. There does seem to be some start up time/delay in doing this, but it seems "pcmrecord" doesn't care or at least times out and eventually retries and picks up at the next available record/decode cycle. BTW, I'm using /dev/shm for all the WAV file writing as not to wear out my SSD.

So there it is in a nutshell, not rock science, but surely a fun science project :) I'll share more as requested and as I make more progress.

73

andyz - K1RA

I'm using

-- Reply to this email directly or view it on GitHub: https://github.com/ka9q/ka9q-radio/issues/47 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

Thanks Jim, this helps a lot towards achieving my goal!

andyz - K1RA

Catching up here.... take a look at jt-decoded.c. It creates temporary .wav files and invokes a wspr, ft4 or ft8 decoder on each as appropriate, then deletes the wav file after the decoder finishes. In addition to the internal logs for each decoder, merged summary logs are kept in /var/log/{wspr|ft8|ft4}.log.

The important thing is that I only have to configure each set of frequencies in the radiod config file, with the output streams going to one of three multicast groups: wspr-pcm.local, ft8-pcm.local and ft4-pcm.local. The decoding daemons don't have to be told the frequencies in each group; they simply process whatever streams they see, in parallel.

Because the decoders in wsjtx and ft8_lib were not written for simultaneous multifrequency execution they did require some minor modifications, which is why I have forks of wsjtx and ft8_lib. The output formats were changed to include full date and time to make them easier to grep in a log, and the ft8 and ft4 decoders were modified to accept a base frequency on the command line. I don't like hacking relatively stable code written by others when I'm not sure what I'm doing, so I need to go back to the authors and see if we could work out some cleaner ways to give me what I need. In particular I'd like to see them all execute in a common directory with file locking to allow them to share logs and databases (e.g., the hash table); having to create a separate directory for each seems clunky.

I'm in the midst of a major rethink of how metadata gets passed to applications. I've overloaded the RTP SSRC (Stream Source) and PT (Payload Type) fields past their breaking points. SSRC currently encodes the frequency (which is assumed to be fixed) and PT encodes the sample rate and channel count but I never intended that as anything but a temporary hack until I can redo it right by having the applications get their metadata directly from radiod's control/status multicast group. Among other things this will allow (but not require) that radiod channels be dynamically created and deleted by the applications themselves instead of predefined statically in radiod's config file. This will of course require all the applications to support the new scheme so I have to tread carefully. But it must be done; the current scheme is too complex and error prone and I'm spending too much time helping debug configurations.

Hi Phil Update on my progress. First, QRA-Skimmer has been running for over 2 months on a Xeon 8-core 4 GHz (Dell Precision 5280) using a RX-888 v1. It running as a binary under Ubuntu 22 as a compiled Python app. I've got 63 unique band/mode combinations decoding at simultaneously from 630m-10m (including 22 & 11m), on FT8, FT4, WSPR, FST4W and JS8CALL. It should also handle JT65, JT9 or other modes that the native WSJT-X 'jt9' command line WAV decoder will support and I'm utilizing. I'm using the 'js8' command line decoder and Python libraries for JS8Call. My decoding results can be seen searching for K1RA-4 on PSKreporter, WA2ZKD WSPR rankings, WSPR.rocks, etc. as compared to my pair of RaspSDRs (K1RA-PI) or pair of STEM125-14 Redpitayas (K1RA). This system seems to hold its own, though I'm convinced the old RX-888 isn't as sensitive on the higher bands. BTW, I never realized how much JS8CALL was being used in Europe on 11m! My other systems don't have the resources to watch as many frequencies.

So for one, I am crontab'ing a restart of my app once an hour, that also restarts kaq9-radio service as the data streams seem to slip more than 2 seconds after several hours of operation and never recovers causing the 'jt9' decoder not to decode any more. This seems to be related to the number of signals I'm trying to decode simultaneously. Those increases as band conditions change during the day, resulting in more activity during 7.5, 15, 60 and 120 second segments. CPU utilization on the even minutes can run 100% for 20-25 seconds, but overall for 4-8 hours the system runs smoothly without slippage. I've 'reniced' radiod and pcmrecord to a higher priority and wsprd to lower and that definitely helped in the early days when I noticed issues.

Let me preface my comments below on my current operations. I'm still not perfectly clear on how KA9Q-Radio utilizes CPU resources based on the radiod.conf and allocating frequencies and modes for streams up front. If one defines say 100 unique frequency/mode combinations BUT only chooses to pull data from a fraction of those say 10%, how much if any time and power being used to keep those over 90% unused streams running in the background? In your current design is CPU utilization any different running 4 mode lines (WSPR, FT8, FT4, XXX) of 25 frequencies per line than 100 lines of one frequency each? You'll see why I ask below in my hack.

As I'm piecing together QRA-Skimmer and not actually rewriting anything from scratch, I first ended up just modifying your pcmrecord to support a '-f' filename option. The scheduler/decoder package I'm reusing is expecting individual WAV file names. I know this breaks your usage under typical conditions, but as everything I've done here is a hack :) and for fun and this was the easiest to get working. Also at this point its not meant to be shrink wrapped, though hope to share if anyone else is interested in trying this.

The scheduler/decoder wrapper I'm reusing expects one to define which and how many simultaneous channel combinations your CPU can simultaneously handle. It also handles band/mode hopping, so if you wanted to cover 100 unique band/mode combinations throughout the day, but your system isn't powerful to do that all at once, it'll time share and tune a stream/channel of the radio and adjust accordingly. I use the scheduler/decoder's config file to auto generate a radiod.conf file with one stream per unique band/mode channel, noting this could also be a band/mode list for sharing (hopping) purposes of the channel. I end up with a template like below, in my case 63 total RX_ lines

[RX_1] disable = no data = F1_rx.local ssrc = 1 freq = 10.0m

[RX_2] disable = no data = F2_rx.local ssrc = 2 freq = 10.0m

Yes, I know you talk about typically running multiple frequencies for a mode on a single stream, but unfortunately without more time and a rewrite that just doesn't fit this other package I'm reusing which wants to spawn a WAV file recorder per channel. I now set and easily get a known named WAV filename out of each RX_ using my tweaked version of 'pcmrecord2' that simply allows for defining a static filename.

['pcmrecord2', '-v', '-d', wavdir, '-f', recfile, '-L', rtime, mcast]

Again whether or not this method is any less CPU efficient, I'm not sure how it compares to yours, but it works. This method and my current app also uses your 'tune' app to share an RX_n stream. Used when band hopping amongst a list of frequencies/modes for one channel.

['tune', '-r', 'rx888.local', '-s', str(ssrc+1), '-f', str(freq)]

Note I keep track of the SSRC as I auto generated the RX_n radiod.conf

I do have an open request for 'tune' as it is not possible to adjust high / low cut from the command line which would be nice as when I dynamically change frequency/mode to FST4W and WSPR I could change to a narrower bandwidth from the 3 KHz for other modes to 400 Hz. For now I have a hack to statically setup radiod.conf and allocate certain RX_n channels to the narrower bandwidth.

[RX_30] disable = no data = F30_rx.local ssrc = 30 freq = 10.0m low = 1300 high = 1700

[RX_31] disable = no data = F31_rx.local ssrc = 31 freq = 10.0m low = 1300 high = 1700

So in short this all works by first defining a Python config file for bands/modes or band/mode lists that are read on app start. In turn that is used to auto generate a radiod.conf file and restarts radiod and renices it. The scheduler kicks off 7.5, 15, 60 and 120 'tune' and then 'pcmrecord2' sessions according to frequency/mode. WAV files are save in /dev/shm, handed off to 'jt9' or 'js8' and then decodes are processed and uploaded to PSKreporter or WSPRnet.

I also have a Telnet server to emulate a simple DXcluster node similar to ReverseBeaconNetwork's FT8/FT4 aggregator service using data from the digital mode decoders. Likewise I have a UDP broadcaster that emulates WSJT-X's ability to share decodes with apps like GridTracker over IP unicast or multicast.

Again I appreciate all your efforts in putting together this SDR package. Its been a fun side project to occupy my (fleeting) spare time. I hope in the near future to put together some minimal documentation from my efforts so I can share it online for others to try.

andyz - K1RA

Seems like a lot of extra complexity, but whatever works. I want to get away from having to frequently modify and restart 'radiod', which is why I added the ability to dynamically create and retune a channel. See my control/status protocol, it's intended to allow lots of different ways, complex or simple, to control and query the state of a receiver channel. One implementation is my 'control' program, which is intended to be interactive. Another is G0ORX's 'ka9q-web' web server (see fventuri/ka9q-web). The 'tune' command is simple, and I think it's what you want for shell scripts. In that vein, your request for additional filter params to the 'tune' command seems quite reasonable, I'll do it. Sorry I missed it earlier.

Oh, I wanted to point out that 'radiod' channels are cheap, so don't be afraid to have a lot of them. That's what ka9q-radio is all about. A typical channel, using the linear mode and putting out 12 kHz samples, only consumes about 0.7% of a typical CPU core; it's actually difficult to measure accurately because it's so low. The only remaining consideration is the network load, which is also low; at 12 kHz mono it's one 480 byte packet every 20 ms; negligible on a 1 Gb/s Ethernet especially if you have IGMP-snooping switches that suppress unwanted traffic, or if you set TTL=0 to confine multicast traffic to the originating machine so you don't generate actual Ethernet traffic at all.

ka9q / ka9q-radio

New QRA-Skimmer Multi Digital Mode Decoder/Reporter Details #47