IQ input isn't decoding and the audio recording is

[janvgils]

This behavior isn't new and I have been scratching me head about this multiple times.

Lets start with some context:

gr-satellites is used with the satnogs integration that is created by @kng. There are multiple examples where I experience that the real time IQ input isn't decoding frames and when I replay the ogg recording it is.

As an example we use the following observation. Here we receive ROBUSTA-1 that uses AFSK 1200 baud AX.25. There is one frame in this observation with enough SNR.

When I use the ogg file I get the following frame output:

# Decoding Satellite 42792 with OFFSET disabled.
-> Packet from 1k2 AFSK downlink
Container: 
    header = Container: 
        addresses = ListContainer: 
            Container: 
                callsign = u'F4KJE' (total 5)
                ssid = Container: 
                    ch = False
                    ssid = 0
                    extension = False
            Container: 
                callsign = u'FX6FR' (total 5)
                ssid = Container: 
                    ch = False
                    ssid = 0
                    extension = True
        control = 0x03
        pid = 0xF0
    info = b"!\xbdC\x83d\xff\xff\xff\xff\xff\xff\x01\x02\x01,\x01,\x01\xff\xff\xff\xff\xff\xff\x01\x00\x01\x00\x0e\x02\xea\x02\xe9\x00\xe0\x01\xeb\x00\xe4\x01\xf7\x02\xf8\x02\xf8\x02\xf7\x02\xff\x03\xff\x03\xff\x03\xff\x03\x00\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x0c\x00\x0c\x00\x0c\x00\x00\x02\xa9\x02\x00\x02\xe2\x02n\x00\xe5\x00m\x00\xe0\x00\xf6\x02\xf6\x02\xf6\x02\xf6\x02\xff\x03\xff\x03\xff\x03\xff\x03\x00\x00\x00\x00\x00\x00\x00\x00\r\x00\r\x00\x15\x00\r\x00\x17\x00'\x00\xac\x0c\x17\x00'\x00\xa2\x0c\xea\x00\x9b\x00\x98\x00\x02\x04\xf1\x02\xda\x03@\x15\xfd\x05,\x01\x03\x15\xcf\x037\x15\xd8\x03\x04\x0e\xea\x03\xfe\x13\x99\x03\xb4\x031\x14\xe9\x01\\\x00\x00\x00N\xbdC\x83d\x00\x00\x00S\xc6C\x83d\x00\x00\x00S\xe7\xc9\x80d\x00\x00\x00S\xf3M\xf5c\x00\x00\x00Nu\xf6\\Y\x00\x00\x00S~\xf6\\Y\x00\x00\x00S3\x82SY\x00\x00\x00N\xe69QY\x00\x00\x00S\xef9QY\x00\x00\x00ws\x0cdR\x00\x00\x00" (total 256)

When I use the IQ file it doesn't provide a decoded frame.

Script used to decode the audio: ogg2sids Script to convert raw to wav: raw2sharp

Command line to decode the IQ recording:

First the direct raw file and as a second option the file after conversion with raw2sharp (this is also use to replay the file in SDR# and or SDR++ to help analyzing the observation)

gr_satellites 42792 --rawfile iq_48000_7690341.raw --samp_rate 48e3 --iq --use_agc --hexdump
***** VERBOSE PDU DEBUG PRINT ******
((transmitter . 1k2 AFSK downlink))
pdu length =          2 bytes
pdu vector contents = 
0000: ff ff 
************************************
***** VERBOSE PDU DEBUG PRINT ******
((transmitter . 1k2 AFSK downlink))
pdu length =          2 bytes
pdu vector contents = 
0000: ff ff

gr_satellites 42792 --rawint16 iq_48000_7690341.wav --samp_rate 48e3 --iq --use_agc --hexdump

(no output)

The IQ recording can be found at the following location: issue-463

When necessary I will provide more information and or details.

[daniestevez]

There is a single strong packet in that recording. This is how the symbols look like when running as

gr_satellites 42792 --rawint16 iq_48000_7690341.raw --samp_rate 48e3 --iq --dump_path .

The symbols look reasonably clean, but there are a few of them with a lot of noise. I don't know why, since the waterfall in Inspectrum doesn't seem to have much interference. Since AX.25 has no FEC, if any of these symbols is wrong, the frame will not decode. Perhaps it's just one symbol that is wrong. Probably it's just a matter of luck using different filtering to get this wrong symbol to flip and get the frame.

For comparison, this is the same plot obtained from the SatNOGS OGG file, decoded as

gr_satellites 42792 --wavfile satnogs_7690341_2023-06-09T19-30-05.ogg --dump_path .

The few noise symbols are not noisy anymore. I wonder why.

[daniestevez]

Can you or @kng give more details about how this .raw file is produced? I'm beginning to think that maybe there are glitches in the raw file because of how it is recorded. Even considering running the .raw through a gr-satnogs flowgraph to try to get an equivalent .ogg and see if that works, but I'm not sure if I'd get all the parameters set exactly in the same way that SatNOGS does.

[K4KDR]

Hello! I have only used .RAW source files of the type that comes from GQRX (i.e., float-32) which are input to gr-satellites as type COMPLEX.

So, I had to look up how a 16-bit signed int. I/Q file could be used and see this note about the TYPE:

... however, I'm not sure how to transition from an FILE SOURCE block of type SHORT to the decoder blocks that would follow.

[daniestevez]

Just to avoid any possible confusion, the .raw file that Jan has shared for this issue is in complex int16 format, and should be used with --rawint16 and --iq with gr-satellites, as I've indicated in https://github.com/daniestevez/gr-satellites/issues/463#issuecomment-1585752038.

[daniestevez]

... however, I'm not sure how to transition from an FILE SOURCE block of type SHORT to the decoder blocks that would follow.

Ah, now I read this again and see that your question may be more on the lines of how to read such a file in a GNU Radio flowgraph. You need a File Source with type set to short (yellow), followed by an IShort to Complex block. I usually set the scale of the IShort to Complex to 2**15 to get an output with reasonable amplitude (less than 1).

[daniestevez]

Doing a 6 kHz lowpass filter and quadrature demod in GNU Radio I see some glitches in the output. I don't know if these can be attributed to the SNR of the signal or if they're caused by lost samples in the .raw recording (which would cause phase jumps in the signal). I now wonder if OGG encoding simply filters these impulses out.

[daniestevez]

No. It doesn't. This is the quadrature demod output saved as OGG in GNU Radio (with the WAV File Sink) and opened in Audacity.

Interestingly gr-satellites does manage to decode the packet from this OGG file (and the symbol plot looks more or less like with the SatNOGS OGG). It also works if I save to WAV instead of OGG (ruling any weird effects caused by OGG lossy compression).

So this leaves me wondering, but at least sets a clear path to compare how what I've done here is somewhat different from what the gr-satellites FSK demodulator does when run in IQ mode.

[K4KDR]

I happen to have a known-good .RAW I/Q file here for testing with a single 1k2 AFSK packet. Just for reference, it decodes very well in gr-satellites using the FILE SOURCE block with a complex I/Q file (32-bit float in this case):

[kng]

the satnogs .raw is a bit odd with the scaling, gr_satellites uses 32768 iirc, but satnogs flowgraphs have 16768. This is the two ways I have been using the .raw in gnuradio in the past (scale 1/16768). This should technically not be a problem when converted to float thou ?

[janvgils]

Thank you all for taking the time investigating this behavior.

One remark so we are all on the same page, this behavior of not decoding an IQ stream also happens when running in real time.

Here is a part of the SatNOGS flowgraph that shows how the IQ file is created.

I will also try to share other examples, one that comes to mind are the BEESAT satellites.

[daniestevez]

I have found the issue. When the AFSK demodulator is run in IQ mode, there is no low-pass filter in front of the quadrature demodulator, unlike in the FSK demodulator. This clearly explains the reduction in performance. I think that I haven't given as much attention to the AFSK demodulator as to the FSK demodulator, since relatively fewer satellites use AFSK. I'll fix this now.

[K4KDR]

Thanks for the update, Jan!

I confess to being confused about the final sample rate of the resulting I/Q file. The SatNogs 'Observation' shows a hardware sample rate of "2.048e6" (common for an RTL-SDR). In your flowgraph, I see a reference to 57.6k (Doppler Compensation block) plus a scale factor of 16.768k (IQ Sink block).

Would you mind terribly outlining how we get from the SDR's native sample rate to the effective net sample rate of the I/Q file?

Thanks so much!

-Scott

[K4KDR]

Dani is correct (what else is new?).

I had tried a BAND-PASS filter with no success, but did not try a LOW-PASS filter. So, another lesson learned.

Thanks very much as always!

[daniestevez]

Just pushed a commit that solves this, by adding the required low-pass filter. With this fix, the frame can be decoded with

gr_satellites 42792 --rawint16 iq_48000_7690341.raw --samp_rate 48e3 --iq

The plot of the symbols also looks good

Note that this only solves a specific problem with the AFSK demodulator in IQ mode. If anyone has encountered a similar problem with the FSK demodulator or in the BPSK demodulator (less likely), where the results are better with the OGG audio than with the IQ file, it would be great to have an example to troubleshoot in the same way.

I'm not going to close this issue just yet, since there was other conversation ongoing, but feel free to close it whenever you want.

[janvgils]

Good morning Scott,

Sorry for the confusion, the flow graph shown is just an example and not this observation specific.

[janvgils]

Just pushed a commit that solves this, by adding the required low-pass filter. With this fix, the frame can be decoded with

gr_satellites 42792 --rawint16 iq_48000_7690341.raw --samp_rate 48e3 --iq

The plot of the symbols also looks good

Note that this only solves a specific problem with the AFSK demodulator in IQ mode. If anyone has encountered a similar problem with the FSK demodulator or in the BPSK demodulator (less likely), where the results are better with the OGG audio than with the IQ file, it would be great to have an example to troubleshoot in the same way.

I'm not going to close this issue just yet, since there was other conversation ongoing, but feel free to close it whenever you want.

Thank you for analyzing and providing this fix.

I will try and find the other examples where I had a similar outcome.

[K4KDR]

Thanks, Jan! Just always looking to learn something new.

[janvgils]

Thanks, Jan! Just always looking to learn something new.

I know the feeling.

And I forgot one other thing, the sample rate difference is, because there are multiple decimation steps in the whole SatNOGS flow graph chain, below the FSK chain as example:

To finally determine the sample rate from the saved SatNOGS IQ file I use a python script find_samp_rate.py and that is used with the satnogs-post script.

I hope this clarifies the confusion.

[janvgils]

I knew there where more examples, here is another one: https://network.satnogs.org/observations/7829368/ this is one of the latest satellites launched by the Soyuz-2.1b / Fregat Vostochny launch on 2023-06-27 11:34 UTC.

The yml used:

name: STRATOSAT-TK1
alternative_names:
   - RS52S
norad: 99151
data:
   &tlm Telemetry:
     unknown
transmitters:
   9k6 FSK downlink:
     frequency: 435.870e+6
     modulation: FSK
     baudrate: 9600
     framing: GEOSCAN
     data:
     - *tlm

During the realtime observation where IQ is used there was no decode, using the ogg file from the observation is able to decode.

***** VERBOSE PDU DEBUG PRINT ******
((transmitter . 9k6 FSK downlink))
pdu length =         64 bytes
pdu vector contents = 
0000: 02 00 1f 02 22 03 00 00 00 00 00 f0 00 00 00 00 
0010: 80 84 1e 00 00 00 00 00 00 00 00 01 00 ec 1e d8 
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
************************************
***** VERBOSE PDU DEBUG PRINT ******
((transmitter . 9k6 FSK downlink))
pdu length =         64 bytes
pdu vector contents = 
0000: 02 00 3e 05 09 18 1d 05 bd 44 a8 64 2a 2b 27 26 
0010: 00 00 00 00 09 0b 08 0b 00 00 00 00 00 00 00 00 
0020: d1 10 4e f0 11 4f 1d fc fb 02 80 05 04 07 10 02 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
************************************

The OGG audio file can be found via this link IQ file is available via this link

[janvgils]

The latest Vostochny launch has a lot of satellites using USP framing, currently the real time decoding aren't producing frames, when the audio file created by SatNOGS is used as a replay input we do get decoded frames.

Example: https://network.satnogs.org/observations/7842384/

Decoded frame based on the audio file:

-> Packet from 2k4 FSK downlink
Container: 
    header = Container: 
        addresses = ListContainer: 
            Container: 
                callsign = u'R2ANF' (total 5)
                ssid = Container: 
                    ch = False
                    ssid = 0
                    extension = False
            Container: 
                callsign = u'RS42S' (total 5)
                ssid = Container: 
                    ch = False
                    ssid = 1
                    extension = True
        control = 0x00
        pid = 0xF0
    info = b'FB\x02\x00\x01\x00\x19\x00\xff\xfd\xff\x81!\x13\xde\x0e0x\xaed\xd6\x9d\x04\x00`\t\x00\x00+\x00\x18\x1f\x00!\xde\x04\x00\x01\x00.\x00\xe3\x15S\x14\xe8\x14\x01\x00\x91\x01\x00\x00\x96\x00\x02\x00#\x00\x00\x00\x00\x00%\xfe\x00\xfe\x06\xfe\x0c\xfe\x000\x00\x00\xd4\x1e%\x9f\x04\x00V\x92\x99ZS\x04' (total 87)

[kng]

I knew there where more examples, here is another one: https://network.satnogs.org/observations/7829368/ this is one of the latest satellites launched by the Soyuz-2.1b / Fregat Vostochny launch on 2023-06-27 11:34 UTC.

StratoSat TK-1 / RS52S, confirmed working live with udp stream on my station 1864, only modified the norad id in satyaml: https://network.satnogs.org/observations/7865723/

[janvgils]

Thanks for the support,

As we discussed this specific example seems to be related to /tmp/.satnogs/grsat_list.txt when a yml file is manually changed this file needs to be regenerated otherwise the Norad id won't be recognized by grsat-wrapper.sh

Well good to know that this example is explained and solved.

grsat-wrapper.sh: https://github.com/kng/satnogs_gr-satellites/blob/main/grsat-wrapper.sh

[daniestevez]

I knew there where more examples, here is another one: https://network.satnogs.org/observations/7829368/ this is one of the latest satellites launched by the Soyuz-2.1b / Fregat Vostochny launch on 2023-06-27 11:34 UTC.

For me, using

gr_satellites sat.yml --wavfile iq_57600_7829368.wav --iq

where sat.yml is the YAML file you shared in this comment decodes 200 frames.

[kng]

I have also looked at the command for getting the wav sox -t raw -b 16 -e signed-integer -r 48000 -c 2 <in.raw> <out.wav> and it looks like the data itself is untouched, only the 44-byte wav header is added. The original .raw and .wav with the header cut off have the same md5sum.

In the satnogs integration there's a known issue when the temporary id's are used, the script only has the TLE for identification and this is usually a followed (guessed) ID, so not easy to get the demod running on those.

[janvgils]

I have found the issue. When the AFSK demodulator is run in IQ mode, there is no low-pass filter in front of the quadrature demodulator, unlike in the FSK demodulator. This clearly explains the reduction in performance. I think that I haven't given as much attention to the AFSK demodulator as to the FSK demodulator, since relatively fewer satellites use AFSK. I'll fix this now.

Daniel,

Trying to demodulate a SALSAT observation https://network.satnogs.org/observations/8391913/ and the realtime IQ isn't decoding and the ogg file replay is.

Is it possible the AFSK demodulator in combination with SALSAT framing need the same change?

The IQ files can be found at this location.

Thank you for all the support.

[daniestevez]

Thanks for sharing.

The problem is that the SALSAT signal is badly designed. In the following aspects:

• The FM deviation is too high (perhaps around 10 kHz). This is what makes a 1k2 AFSK signal occupy more than 20 kHz.
• The packets are not helpful for closed loop symbol clock recovery. There is a long burst of zeros at the beginning of the packet, which doesn't help and probably makes things worse (by upsetting the loop more than what random noise would), and then a very short training sequence of 0101 (only 24 symbols) before the packet starts.

The situation with the OGG recording is not great either. There are many strong packets in this recording, but for me the OGG recording only decodes one packet.

The very high deviation is what prevents the IQ file from decoding at all. The AFSK demodulator assumes a deviation of 3 kHz to apply Carson's rule and lowpass filter the signal before quadrature demodulation. This destroys this particular waveform (I plotted waveform.f32, with --dump_path enabled, and the result was awful). To see what happens I have manually increased this deviation in the code (there is no command line argument or YAML file parameter to do this) to 10 kHz. This gets a clean waveform.f32. However, symbol clock recovery is not great, due to the short training sequence. By playing with the loop bandwidth I managed to decode some packets. For instance,

gr_satellites /tmp/SALSAT.yml --iq --clk_bw 0.1 --rawint16 iq_48000_8391913.raw --samp_rate 48e3 --dump_path .

decodes 3 packets.

Another point is that the deviation of the AFSK tones is 300 Hz, not 600 Hz. The lower tone is at 1200 Hz, which can easily be seen in the spectrum of the OGG file. I don't know where this deviation of 600 Hz came from. It seems inherited from the SNET satellites, and maybe it's also wrong for those. I'm going to fix the AFSK tone deviation in the YAML now, but for the FM deviation I'm not sure what to do. Maybe a command line parameter to handle these corner cases would be good.

[janvgils]

Hi Daniel,

Thanks for the extensive explanation, I will update the yml and see if we can have better decodes with the altered yml file.

[daniestevez]

Regarding the problem with SALSAT (it used a very high FM deviation but gr-satellites was hardcoded to use an FM deviation of 3 kHz), I have now added an --fm-deviation command line parameter and an fm_deviation SatYAML field to set the FM deviation when needed in 2fb45c2219627517ea046756b5a35fe9860ce554.

[daniestevez]

I think I'm going to close this issue now, since at the moment there are no open cases of "IQ doesn't decode but OGG does". Whenever someone finds an instance where some kind of recording isn't decoding but it should, then a dedicated issue can be opened.

[janvgils]

Thanks for this new option and I see after some tests that this new option indeed makes it possible to decode SALSAT observations.

I will close the issue