search

projecthorus / radiosonde_auto_rx

Automatically Track Radiosonde Launches using RTLSDR
GNU General Public License v3.0
477 stars 124 forks source link

Improved FSK Decode Chain #104

Closed darksidelemm closed 5 years ago

darksidelemm commented 5 years ago

This is a tracking issue for improving the FSK demodulation signal processing chain.

My intent is to make use of the FSK modem from David Rowe's codec2-dev repository (see here: http://svn.code.sf.net/p/freetel/code/codec2-dev/src/fsk_demod.c ) This modem can accept either real samples (i.e. USB demodulated data), or raw IQ data straight from the RTLSDR. The caveat here is that the input sample rate must be an integer multiple of the baud rate in use. It can output either:

Some changes are needed to the fsk_demod command line utility for it to be usable in this project. The main one is the ability to manually define the FSK frequency estimation limits. Since I hope to feed fsk_demod with a large amount of bandwidth from the SDR, I want to be able to define a window where the sonde is expected to be, based on the results of the scanning and detection step, but with enough window range to allow for some drift in the sonde and/or the receiver.

This change should provide the following:

The initial steps that need to be performed to make this a reality is:

rs1729 commented 5 years ago

Just looked into codec2/fsk_demod and fsk_mod, but it didn't let you use negative frequencies... In the description I often read the word "magic" or that he doesn't know why it works (e.g. MSK), that's a bit odd. I think it uses the following approach: if you have modulation index h=1.0 and shift to one of the FSK frequencies, the linear combination of samples over one symbol (M samples) will be either zero or M times absolute value squared of one sample (constant amplitude). For other modulation indices the sums could be even further apart, however for shaped pulses (GFSK) the results will not be as good I guess. You can derive the modulation index for a particular radiosonde type from a clear IQ-recording, but maybe it is even better to test which index-setting gives the best bit-separation or best decoding performance. When integrating (same for FM-samples) most important for me is synchronization and also time resolution/sample rate. Then integration is almost as good as a matched filter, for rectangular pulses even the same. codec2/fsk_demod seems to do a lot of work on sync. On the other side, the modern radiosondes are stable enough, so it works if you have good sync at the beginning of the frame. (The rtl-sdr is also good enough.) And upsampling/interpolation is not so expensive. Just made a short test on my own with IQ-recordings and non-coherent decoding since I tried differential/FM-demod of IQ before, but results are similar to FM-demodulation. The FM-demod & integration is maybe more universal for most of the radiosondes and easier if you want to serve different kind of receivers.

I think if you use IQ-samples it is more interesting to use the whole bandwidth, look for peaks and detect signals, and then decode in a separate thread in parallel, so you can use one rtl-sdr for several radiosondes.

A few remarks: If you want to have something like SNR for the FM-signal, you could look at how good the correlation-detection is. For the same setup you get some kind of signal quality indicator that way. The Reed-Solomon decoder also gives you the number of corrected errors, if the signal quality is low. Could also tell how reliable the DFM-fec is, when you have many corrected bits. Then for frequency offset on FM-audio you could use the mean value, works good for all the Manchester coded radiosondes (cf. M10 --dc; for M10, sample rate is the bigger problem). The preamble of the newer RS41s is also useful for that. (The M10 false positive problem could be solved when comparing with RS41-header correlation. The M10 does not really have a header, but you could also look at the high-frequency carrier, from where the signal deviates down.) Apropos Manchester code, if you can integrate over two symbols/raw bits, that will improve SNR alot. I think Viproz was talking about the same. Soft decoding could be useful for Reed-Solomon codes (w/ erasures), i.e. RS41 and RS92; so far I was a little bit disappointed when trying to automate this, it was much easier to correct suspicious bits/bytes by just looking at the data. I believe soft decoding is most useful for LMS6 (Viterbi/RS). Just flipping all the bits can be alot of work very fast (and I wouldn't call that "fec" by the way). Furthermore the Hamming-Code of DFM can give you the wrong codeword very easily because the codewords are close together; it is not the best code when you have so much noise, an additional checksum or crc would be fine. And for M10 ecc would be great, the frames are so short anyway.

darksidelemm commented 5 years ago

Hi! Great to hear from you! Your work in this area is amazing, and it's great to get some comments from the guy that made this project possible!

The frequency estimation range for fsk_demod is currently fixed based on the sample and baud rates, but that's fairly easy to change. I guess the main reason I wanted to switch to it was to lessen the criticality of being spot-on frequency, which I've found is an issue for the RS41s, and probably would be for the 1680MHz sondes common in the US too. I guess there's improvements to be made with better detection of the sonde frequency which could solve that as well, but I kind of like the idea of being able to drop a window around the transmit frequency and let the modem figure it out.

I definitely want to do some proper performance testing, where I record a 'clean' radiosonde sample (as IQ) and then add noise to produce a set of samples with known SNRs (at maybe 0.5 dB steps). We can then run these through various demodulator chains to look at what performs best, and work out how to optimise for best performance.

Anyway, the aim of this issue was to provoke discussion, which it has!

rs1729 commented 5 years ago

Thank you! When I looked at my first radiosonde audio recordings, the spacing between the zero-crossing were the first and easiest thing to try to extract data. Soon I realized that not only one bit breaks down if noise adds unwanted zero-crossings... And for error correction codes that's not good. Integrating samples is not more complicated if the sync is right, so far it works for me. Frequency offset is another problem especially with automated detection or if you don't have a controll-FFT-display (and unknown ppm-offset of rtl-sdr). After my first steps with these scan-scripts I didn't work on that anymore, but it is good to see someone working on auto-detection. For me a simple high pass filter helped a little, but performance was not very good, and with auto-detection the interesting signals are often weak signals. If I center a signal in the spectrum-view, it is not a real problem. Only for the DFMs (and also M10) you have to adjust after from time to time. I think the LEO-sat guys have more experience with drifting signals.

I don't know if you or Vizproz use the old rs_detect tool from the scan-directory? It doesn't need much computations, but has the known problems of the zero-crossing-demodulation, so a low pass filter is recommended. Maybe the dft_detect-tool works better, but the FFT-correlation needs more resources. (It does not have the return values of rs_detect yet.)

IQ-data is much more interesting, but it is also more work. You have to address the different modulation parameters for different radiosondes more, but in principle you should do better with the IQ data. If you look at FM-demodulated noise it gets pretty loud, when you differentiate the noisy sequence of samples. It is interesting to compare the RS41-signal (with the noise gaps) in IQ and FM; also in IQ you get the SNR easily. And with sdrsharp e.g. you can look at the spectrum and in raw mode still decode the IQ data over stereo-audio. But the real benefit would be the simultaneous decoding, like in sdrconsole or if you do that with the rtl-sdr-tools and process the data in threads. Or on limited machines sequentialyl decoding several frequencies in the bandwidth would also be fine. With IQ-data the uniform preamble of the RS41 (if you detect it) helps you determine the frequency offset (use FFT, e.g. like in rs41iq.c (it is only a quick-and-dirty-test-version), I limited the max frequency offset and got the impression that sdrsharp applies a high pass filter to FM-demodulation... and chops raw-audio if gain is high?), but of course it would be best to have a more general approach. And when you have sync and also phase-tracking, you could do coherent demodulation, but with the gaussian pulses it is still a bit of work I believe.

rs1729 commented 5 years ago

re: detection/decoding/bandwidth, your discussion with viproz on irc. sample rate vs bandwidth are just different views, time vs frequency. The rtl_sdr is sampling in time and the data can show you frequencies up the frequency the sampling theorem describes (with iq/complex-samples it is 1:1). You could do the following (if peak detection/frequency offset is solved): If you have a high sample rate/bandwidth, you shift the signal of interest to f=0, decimate to a sample rate you can handle and that gives you enough time resolution, (detect frequency offset) and then put a low pass filter around your signal. If you use zero-crossing-demod, the low pass filter should be quite narrow, if you integrate your samples, this averages your signal, so your filter needs not to be so narrow (narrower filter could distort symbol boundaries). If you only want to detect a signal and try to correlate with known headers, the low pass filter is even less important. So maybe a sample rate of 48-60kHz is ok to work with and a low pass of 12kHz for the signal (24kHz for M10). Now you can process the iq-data, demodulate fsk directly or via fm-demod. If you use a fm-output with the sample rate that just covers the signal, it is filtered, but it would help to upsamle/interpolate, so you get a better time resolution for processing the samples, and I guess you lose a bit of processing information that way around and if the signal is freq-offset, you lose more. I don't know csdr, but sounds like rtl-fm, what viproz described. So for rs_detect and dft_detect you still want a sample_rate of 48kHz, but a low pass filter reduces the noise outside of your signal.

Viproz commented 5 years ago

Hey thanks for the input I didn't know you were on #highaltitude !

So I'm looking at csdr for the fact that we could split the raw IQ data into multiple streams and decode multiple RS at the same time that are in the same ~2GHz frequency range but I approach the problem the same way we've been doing it, by decoding the signal in FM and then doing the RS decoding.

For detecting the RS we are using rs_detect right now, I've tried to code something to detect more frames by integrating at the correct samples per bit and so far it's working really well for M10s, I can detect a lot more frames than with the method currently used for detecting M10 headers (so 0 crossings) but it might be a bit computation intensive, even more so when adding all of the others RS headers and samples per bit.

So maybe a sample rate of 48-60kHz is ok to work with and a low pass of 12kHz for the signal (24kHz for M10). Now you can process the iq-data, demodulate fsk directly or via fm-demod.

I don't understand this, how can you filter the IQ signal before FM decoding it ? You decimate it by 50 to get to 48 kHz but then how do you filter out to get only a bandwidth of 22 kHz while still maintaining the sample rate ? This is the biggest thing I don't understand, I need to do some reading as to why I can't get a 96 kHz sample rate on a 24 kHz bandwidth, or at least I don't see anything to do it. Edit : I think I missed firdes_lowpass_f, I don't have access to my pi right now, I'll test as soon as I can

For the drifting of M10 I had another idea which might be applicable to DFM too, it is to not low pass filter the FM decoded signal, this way we have an offset from the 0 (we can't use 0 crossing then) and with the M10 we know there are going to be the same amount of 1 and 0 (Manchester encoding) so if we just average the signal we get the offset directly and this offset reflects the center frequency the RS is emitting on so we can feed it back to the FM decoder to select exactly the right frequency and maybe adjust some bandwidth parameters to reduce noise.

rs1729 commented 5 years ago

Sometimes I read the logs...

If you can read the IQ-buffer from several threads, in each thread you shift (the samples) to the frequency of interest and decimate/filter and demodulate/decode; shifting is just multiplying the samples by exp(-iwt) for some frequency w and you get your stream to work on. You reduce the bandwidth/sample rate to reduce the work load.

I put another detect-tool in the repo, scan/dft_detect, that uses correlation to find the headers, like the RS/demod/_dft... do. It also shows how good a certain header was correlated. These demod/dft-decoders integrate the samples like the old ones did with option -b (-b2 for manchester), but they find more headers and sync more accurately. However a sample rate of 48kHz is recommend. For me it works much better, but may depend on the receiver. I sync only at the frame-start. I have seen some M10-recordings, where the zero-crossings were better, as if the timing was not right; sometimes it is enough to upsample. For M10 the dc-offset is also important, m10dm_dft has a --dc option that does exactly what you say, one of the reasons to use manchester code is that the average is 0 in small windows already. It is also possible to use the DFT-data for dc-offset, if you use DFT for header-correlation anyway.

You can filter IQ data the same way as FM-data. A low pass filter doesn't change the bandwidth, it only reduces the higher frequencies. If you have IQ data with sample rate/bandwidth 48kHz and put a low pass filter of 8kHz around the center, the samples are changed in a way that when you look at the FFT-display, the higher-frequencies are attenuated. If you decimate the bandwidth to 8 kHz, you would also have reduced the higher frequencies. But if you are not careful, some higher frequency leftovers are visible in remaining samples, so you need the low pass filter somewhere. Then if you upsample/interpolate to 48kHz for better processing, you would not have the higher frequencies on the FFT-spectrum. Maybe you play with gnuradio or somthing like sdrsharp, where you can record the filtered raw IQ data and play back. You can even use sox for filtering and resampling IQ-data. Maybe the confusion comes from what you mean by "bandwidth". Do you mean the sample rate or do you mean the low pass filter of a certain bandwidth that some sdr-programs show you? Then there is also the bandwidth of a signal.

Just uploaded a test version for (non-coherent) iq-decoding of rs41: RS/demod/rs41dm_iq.c and the other _iq-files. As far as I tested it works better then the FM-decoding for weak signals. But it needs the right IQ-recordings I guess and many different settings, so I don't know how universal it will be. The FM-demodulation makes several things easier, you don't need to know the frequency deviation. But the differentiation usually make a time series less smooth, integrating does just the opposite. With the iq-decoder, integrating "continuously" gives you a signal similar to the FM-signal, but for symbol/bit-decision you can concentrate on the inner parts of the symbol. Anyway it looks promising. But I guess there is no free lunch, weak signals just need more attention.

Viproz commented 5 years ago

So I had noticed that I was decoding M10s a lot better with audio gotten from SDRSharp than when letting the SDR use rtl_fm so I've done some testing with csdr to try and figure things out. I have a good recording of some rtl_sdr output taken with a center frequency of 401.5MHz, the M10 is at 401.0MHz.

All tests were done doing ./m10 -b -b2 -R filename.wav, -b and -b2 for the two alternative methods and -R to get the number of decodings.

The 48k non filtered sample rate is probably influenced by the zero crossing method I'm still using for the header detection, at higher noise levels it's probably an issue. Otherwise 48kHz bandpass filtered really seems to stand out as expected, I'm guessing it would have a big positive impact on other RS that have shorter bandwidth and so smaller sample rate with rtl_fm. My conclusion - as we already knew - is that rtl_fm is not the right tool for the job, csdr seems to be really powerful but not compatible with every system but finding a better tool could improve overall performance by a big margin.

darksidelemm commented 5 years ago

I agree - something with more flexibility in filtering would be a better solution.

@csete (Gpredict, GQRX) has a SDR-agnostic (kind of) demodulator utility that may be appropriate, however he hasn't released it to the public yet. I've asked him when he plans on releasing it, as it may be a good solution.

rs1729 commented 5 years ago

The last example is how I would do it. The bandpass filter (lowpass if you shifted the signal to 0) is like the grey filter-window you select in sdrsharp when demodulating. I think, before demodulation, sdrsharp decimates by a power-of-2 factor down to a sampling rate of at least 32k (hence the FM-filter/"window" is at most 32k wide). E.g. a 1920000 Hz/sampling rate baseband is decimated by 32 to 60k, 1024000 by 32 to 32k. (You can see the IF-bandwidth e.g. in the IF-recorder-plugin.) The decimation also filters (anti-aliasing), then you have a sample rate/bandwidth you can process easier. The filter before FM-demodulation lowers the higher frequency parts, otherwise the FM-demodulation that outputs the change in frequeny, would be very nervous. You are only interested in frequencies that belong to the signal.

In rtl_fm the decimation is the only lowpass filter, so you decimate to a lower rate and then interpolate back with sox to 48k. Maybe the 48k raw output of rtl_fm is more useful if you would have a good lowpass filter for the IQ samples. The sox lowpass filter falls off very slowly when I tested, it probably isn't the best filter for that purpose. Another thing is the center frequency. For r820t it should not be so important, but in sdrsharp if you look at the IF-FFT when you are at center frequency you still see some distortion. I don't know how much this affects the demodulatino. I don't believe rtl_fm processes the signal off-center, it tunes the rtl_sdr to the desired frequency so it does not have to shift anymore. I think rtl_fm was ment to be efficient and lightweight, for weaker processors, so maybe that's why it tries to minimize the processing steps.

I never tried csdr, but it seems to have all the functions you need. Decimating 2.4M by 50 to get 48k is exactly what I would want. The math operations are not complicated if you take the complex-math library, but this is also very slow. Maybe it is optimized in a way that is less portable. If you want to test the single steps of csdr, you can view the IQ-data after every step in sdrsharp (output the right wav-header using sox).

rs1729 commented 5 years ago

Using only tools already available, with tee one could split the rtl_sdr stream and decode several radiosondes at the same time. Here are two examples: https://www.youtube.com/watch?v=RRvfkKBbkBg 1) with pre-recorded rtl_sdr IQ-data 2) live decoding. (gqrx to show what's on the spectrum.)

The last stdout could be split again for a third signal, and so on. (Perhaps "duplicated" is the better word. If you would read the same fifo/pipe with two readers, it each byte would go to only one of the readers.) I think it would be better, if multiple readers could read the same buffer, a device like /dev/dsp, or a dedicated demodulator.

darksidelemm commented 5 years ago

This is a complete project in and of itself - a server that can be centred on a chunk of spectrum, and can spawn multiple narrowband demodulators off the same source data. I would guess communications with the server would be through some kind of network interface, where a client can start a demodulator and receive samples. I know similar has been discussed for used in the SatNOGS project, and it sounds like a fair bit of work to do it properly. I know @csete (GQRX, Gpredict) has had similar ideas.

Definitely a separate project I think, which has many uses other than just auto_rx.

csete commented 5 years ago

Hi,

I know similar has been discussed for used in the SatNOGS project, and it sounds like a fair bit of work to do it properly. I know @csete (GQRX, Gpredict) has had similar ideas.

Indeed, a multi-client sdr server is in the pipeline, although finishing a lightweight replacement for gqrx and the command line rx/tx is my first priority right now.

rs1729 commented 5 years ago

I changed the DFM sign in dft_detect to DFM-09, since I want to use it with dfm09dm_dft. I put a script and fft-scanning together in RS/scan/, a first example to scan and decode multiple radiosondes with rtl_sdr: https://www.youtube.com/watch?v=yN9OUlrCcYY Just a PoC. For M10 the signal centering is not ready, the carrier on one side shifts the center frequency. And for imet1rs and imet1ab the scanning bandwidth has to be wider. I think rtl_sdr produces mirror images easier than sdrsharp. Different signals are decoded in separate terminals. If you output everything to stdout, the frames could be mixed together, so the communcation between writers and reader would have to be synchronized/ put in different time slots.

darksidelemm commented 5 years ago

This is definitely a cool approach, but it would need some kind of way of re-adjusting the SDRs centre frequency if a new sonde was found outside of the SDRs bandwidth, to optimize how much can be decoded. You're still kind of at the point of needing 2 SDRs - one to receive as much as it can, and another to do the search over the radiosonde band looking for peaks.

rs1729 commented 5 years ago

Yes, there are different possibilities. You could choose short intervals of a few minutes or so and re-scan. Or do it all in one program, reading the sdr into a buffer that can be scanned more frequently and decoded. And if a decoder runs out of signal, it could report and close. I want to try power-of-2 decimation, low pass filtering is not a problem, for poor signals I use it anyway, the csdr-fm is not enough. However I don't care so much about optimizing, want to use the complex data type, that is probably quite slow. When forking/threading in my own program probably I could control the communication between different decoding tasks easier. Don't know how it would perform on a raspi.

And frequency-offset correction is on the list. A simple average/dc of the fm-signal is enough for small offset. If you want to track more drift, you have to report to the iq-frequency-shift.

darksidelemm commented 5 years ago

This is definitely where the server-client model that @csete has in mind will help. Currently his nanorx software (which I am intending on replacing rtl_fm with once he releases it!) can emulate hamlib, enabling the RX frequency to be adjusted during reception. If the demodulators can produce an estimate of frequency offset (noting this will be dependent on RX deviation settings I think?) then I could add some code to close that loop, and based on some threshold, re-tune the receiver.

rs1729 commented 5 years ago

Yes, the more you go into the details, the more it becomes a project on itself.

Do you want to tune the RF receiver? If you tune the RF receiver, all signals in the band are affected. Maybe it is better to leave the RF receiver tuned to a constant RF frequency and provide the baseband/IQ-data to "virtual" receivers that tune to a (relative) frequency of interest in that baseband and decimate, and then you fine-tune/frequency-track these virtual/logical receivers before demodulation and signal processing.

darksidelemm commented 5 years ago

So I've been doing some testing of the various demodulators and detector code using some scripts I've developed with help from David Rowe.

So far i've been using samples of RS41, RS92, DFM09 and M10.

I'm still continuing with my investigations, but so far I've found the following so far:

I'm going to continue onwards to figure out the 'optional' demodulator settings for each sonde type. This will help in working out how best to drive rtl_fm/csdr/nanosdr for each sonde type, and what the best compromise setting is for detecting sondes. The samples I'm using and my testing code so far is here: http://rfhead.net/sondes/samples/sonde_snr_testing_20190223.tar.gz There's a README.md with some information what's required to run the scripts to generate low-SNR samples and process them.

Some output showing the performance differences between rs_detect and dft_detect here: https://gist.github.com/darksidelemm/ba5ee2d81b4282a743c4ec4687443386 These are both run using the same source files, and the same demodulator chain of: cat | csdr fir_decimate_cc 4 0.005 HAMMING 2>/dev/null | csdr fmdemod_quadri_cf | csdr limit_ff | csdr rational_resampler_ff 2 1 0.005 HAMMING | csdr convert_f_s16 | sox -t raw -r 48k -e signed-integer -b 16 -c 1 - -r 48000 -t wav - highpass 20 2>/dev/null | <rs_detect, or dft_detect>

darksidelemm commented 5 years ago

After a bit of digging around the rtl_fm codebase (trying to figure out if I could either strip out the FM demodulator, or feed in samples to do some quantifiable tests of it, I discovered that it has a 'raw' output mode. It'll quite happily produce 96 kHz (or whatever) IQ output, in complex-signed-16bit form. This can be fed into csdr fairly easily, with some conversions.

As an example, a RS41 demodulation chain: rtl_fm -f 402500000 -M raw -s 96k -g 49.6 | csdr convert_s16_f | csdr fir_decimate_cc 8 0.005 HAMMING 2>/dev/null | csdr fmdemod_quadri_cf | csdr limit_ff | csdr rational_resampler_ff 4 1 0.005 HAMMING | csdr convert_f_s16 | sox -t raw -r 48k -e signed-integer -b 16 -c 1 - -r 48000 -b 8 -t wav - 2>/dev/null| ./rs41ecc

I could possibly do the decimate-by-8 step in rtl_fm, but I'm not sure how much I trust its antialiasing filtering. Note that i'm decimating tightlyto give a 12 khz bandwidth into the FM demodulator. From my testing this seems to perform best for RS41s.

The same thing using fsk_demod from codec2-dev: rtl_fm -f 402480000 -M raw -s 96k -g 49.6 | ../bin/fsk_demod --cs16 -b 1 -u 45000 2 96000 4800 - - 2>/dev/null | python ../bin/bit_to_samples.py 48000 4800 | sox -t raw -r 48k -e unsigned-integer -b 8 -c 1 - -r 48000 -b 8 -t wav - 2>/dev/null| ../bin/rs41ecc --ecc --ptu --crc

Note that i'm offsetting the RX frequency to put the signal within the frequency estimation window. It can drift around anywhere within that window and still be demodulated. From my testing this demod chain performs about the same as the tightly filtered FM demodulation chain, though with the benefit of tracking frequency drift.

Both of the above examples run fine in real-time on a Raspberry Pi 2, so I'm thinking this may be a reasonable approach moving forward, and would provide a lot of ways we can tweak things to improve performance. I'm not 100% happy with using csdr, and i'd prefer something that's a bit easier to build cross-platform (CSDR does build under OSX, with some makefile hacks, but its a pain, and good luck under Windows). @csete 's NanoRX might be a suitable option for this, when it's release - however it will need customization of FM demod deviation to get the best performance.

For now I might spin up a testing branch which uses csdr as the FM demod to have a bit more of a play.

rs1729 commented 5 years ago

@darksidelemm, rs41ptu.c has now a --bin option, reads 0/1 bit per byte ('0'/'1' also possible). Works with your bin-sample. So there are always 4800 bits coming from 1 second, right? When there is no signal, there are still bits, that's good, since the frame length can vary (aux-data). So it can be processed the same way as the audio-stream, only the samples_to_bit processing is not necessary.

rs1729 commented 5 years ago

The SNR-tests are really good to compare results, though perhaps one should generate several samples for each dB-value for more reliable statistics... But this aside, I don't know if it is csdr-fm when the filter is too wide (once I have seen this with sdrsharp as well) but for weak signals you get these spikes in FM. So I tried to filter these out a little bit. For M10 at least this improved my results, I think it was 16[OK]/106[NO] frames for 9.5dB and 7[OK]/96[NO] for 9.0dB. (decimation by 2 and band filter -0.2 .. 0.2 at 48k) The rs41-samples were different, there it didn't really help. I have some DFM-recordings that also had these spikes. There a lowpass after FM was helpful. Some simple spike-filtering is easy to implement, though I don't know if it should be applied by default, and the parameters depend on e.g. FM-gain.

darksidelemm commented 5 years ago

Agree that we need more packets to get better statistics out. I guess I could just repeat the input file a few times in the generate_lowsnr.py step. I was trying to avoid having to supply a huge base sample file!

rs1729 commented 5 years ago

Oh, I didn't mean the script. I was running out of disk space already... So maybe optional, but for me it's ok.

darksidelemm commented 5 years ago

I've got an experimental branch in my own fork of auto_rx which is using fsk_demod as part of the signal processing flowchain. It's working, but it doesn't handle interferers (with a power > sonde signal) in the passband well. Then again, neither would a FM demodulator. Still, something that might be fixable with more tweaks to the frequency estimator.

I currently have the modem outputting stats (Eb/N0 estimate, timing error, eye diagram and FFT snapshots) to a file, which I can then plot live: https://imgur.com/a/qh3LaE6 It is quite cool to watch! The Eb/N0 estimate (or perhaps a scaled version of it) would be extremely useful to have available in the web-GUI, as a way of debugging a station.

rs1729 commented 5 years ago

@darksidelemm I was testing your SNR samples, the dfm09 gave bad results. Looks like some dfm09 complex-float samples have absolute value > 1, when converting to 16bit signed, they are wrapped around. Scaling to 0.5 amplitude fixed this, though I'm not sure if the sequence was not cropped somewhere.

The other samples look ok, max-values between 0.18 (rs92) and 0.34 (rs41, m10).

With rs41 and --ecc2, if you get all frames OK (119, first and last are not complete), you can count number of byte errors vs number of bytes in the frames, it's like bit error rate.

Then I tested reading bits, 1 bit per byte. If the decoded bits have errors, it might be a problem for header detection, you do not get a score like correlation. If you allow a few bit errors, dfm can detect a flipped header at wrong positions. The noise between rs41-frames produces random bits, though it seems unlikely to get close to the header bit sequence.

darksidelemm commented 5 years ago

You are 100% right on the cropped samples (after type conversion). I found this out last weekend, which is why normalise.py exists (https://github.com/projecthorus/radiosonde_auto_rx/blob/testing/auto_rx/test/normalise.py).

I just haven't gotten around to re-uploading the samples! The code to generate the low-SNR samples does normalisation to +/- 1.0 prior to saving, so all the low-SNR samples generated from the original float samples should be OK (as theres no type conversion occurring in that code).

I'll try and re-upload the normalised base samples tonight.

rs1729 commented 5 years ago

Just wanted to add, comparing/xoring the correct high-snr raw frames with the non-corrected raw frames and counting the bit errors is probably better, though a little more work, but works for the other radiosondes and to lower snr. (and for rs41, 1 byte error could 1..8 bit errors) I think I got about 0.65dB above theoretical non-coherent 2-FSK, ber 0.0107 at 9.5dB, 0.0065 at 10.dB, rs41fm. IQ-decoding decoded also all the 9.0dB frames, though 0.022 is 0.9dB worse then theoretical? (Though 1 byte error means up to 8 bit errors, so it's only an upper bound, I guess.) Narrow filter improved both methods, -0.7..0.7 of 48kHZ for rs41.

If you update the script, maybe you should use for all the manchester-coded rs92, m10, dfm the same settings, either symbol or bit rate?

darksidelemm commented 5 years ago

That's definitely a good idea - we have a known bitstream, may as well make use of it...

I'm still not sure whether to use the raw baud-rate or the post-manchester bit-rate for the manchester-encoded sondes. I'm not really sure you can counter manchester encoding as any kind of coding gain - it's it more a degradation in performance as you need to correctly receive 2 symbols in a row?

I kind of like the idea of using the bit-rate at the entry-point to any other FEC (hamming, RS...) as the reference point for comparison - so for RS92 that's 2400 baud. For M10 that would change it to 4800? I just want to be consistent across all the sonde types - at the moment I'm not.

rs1729 commented 5 years ago

It depends a bit on the receiver, if you can make use of the Manchester coding. If you know where the bit (consisting of the two symbols) starts, you can add (the samples of) the first and subtract the second, so you integrate/match over two symbols, but instead against a rectangular shape you have first half up, second half down. It is the same as integrating each symbol and comparing both results. So if your decoder outputs only digital values for each symbol/raw bit, you loose that information. If you decode every symbol independently and have a symbol error, how do you decide, which bit is it? So it is better to consider both symbols together in the first place.

First I would rather look at the symbol rate (baud) and take the Manchester code as an additional feature that can improve decoding. However if you do it right, it is probably comparable to a signal with half that rate, so the Manchester radiosondes would look better than they are, the actual information transmits slower. But then there is also the ecc-rate that could be considered next, all the parity bits also help decoding, althoug at a later stage.

Anyway, it should be the same rules for all the Manchester coded radiosondes.

darksidelemm commented 5 years ago

Agreed - need to keep the rules consistent. Is it just RS92 and M10 that are manchester encoded?

rs1729 commented 5 years ago

RS92, M10, DFM06/09. (M10 uses differential code, thus polarity not a problem, but a bit error would affect also the next bit.)

https://github.com/projecthorus/radiosonde_auto_rx/issues/122#issuecomment-469061442

darksidelemm commented 5 years ago

Oh interesting, I didnt realise DFMs used it too! So the actual user bit-rate (prior to FEC) for those is around 1250 bit/s?

rs1729 commented 5 years ago

Yes, it is slow. The bandwidth is even a bit wider than rs41/rs92 (the DFM modulation index is more than twice I believe, minimum in center), so if you see the signal on the spectrum, it decodes at a lower signal level (It's not the ecc, Hamming-ecc is less performant than Reed-Solomon).

You could consider the two polarities DFM-06/DFM-09 also as the two Manchester versions.

ah, ok, rs92-sgp is also 8N1, again lowers the bit rate.

And as remarked earlier, as you consider relative levels for Manchester, the frequency offset is not that important, only maybe for header detection.

rs1729 commented 5 years ago

Sorry, I used the old generate_lowsnr.py from the tar-file, there was no normalizing... ok, with the new samples rs41 didn't change much, but dfm. I counted bit errors now, and when considering only baud rate, the dfm was below the limit (so maybe you should take baudrate/2 for manchester), with bit rate it was 3db higher. Although the higher frequency because of Manchester is not really the same as the signal without Manchester and half the baud rate instead (moreover it would have smaller bandwidth).

Although there might be another issue, so I'm not sure about the numbers. For rs41, Reed-Solomon corrects bytes (doesn't matter how many bits in that byte are in error). However at 9 dB, if there is a byte error, there is only about 1.06 bits in that byte wrong. But in total there are 8 times as many bits as bytes, thus when counting byte errors the upper bound is not very sharp. Counting bit errors now, no error correction needed, and then I get the following: 119 frames, 320-8 bytes per frame (not counting header), 8 bits per byte. bit errors: 8.5dB: 0.0047 9.0dB: 0.0028 (=836/297024) 9.5dB: 0.0015 10.0dB: 0.0008 (FM-decoding!) Isn't this 1.2dB below the limit? Perhaps the noise/variance estimate is not so accurate? (What about the "intrinsic" variance of the signal?) Also the dfm has such a high variance.

Didn't do this before, maybe should have done this before, really helps evaluating the decoders.

darksidelemm commented 5 years ago

I do need to re-visit the checking of variance. At the moment I'm literally just calculating the variance over the file (or over bits of it, in the case of the RS41 and M10). This should be OK, and I'm also fairly confident now of how much noise is added (based on this analysis).

There's definitely going to be some degree of error in the results due to the non-infinite number of bits measured, but 1.2dB does seem a bit much (without any FEC anyway).

Are you counting bit errors based on comparison with the high-SNR file? That would definitely be the best approach to take, and is essentially the same approach I took in the analysis linked above.

I wonder if we're getting a slight improvement over theoretical because we're comparing against non-coherent FSK demod performance. Is a FM-demodulator based FSK demod really non-coherent? Or does the PLL action of the FM decode make it closer to a coherent FSK demodulator (which has about 1 db better performance)? Not sure of the answer to this.

rs1729 commented 5 years ago

Yes, the high-SNR frames give me error-free frames. Then I compare to non-ecc decoded low-SNR-files. Discrete sampling (time and amplitude) should it make even worse.

rs1729 commented 5 years ago

Added m10mod.c. FM-decoding is the same as for m10dm_dft, so with the spike-option it is almost as good as viproz, but without 2-pass, i.e. comparing to bytes from previous frames. Though some normalization is useful or spike removal (lowpass is not as effective as in case of dfm), since m10 (as dfm09) show these spikes at least in the SNR samples; not sure, if the unstable amplitude is one reason (didn't seem as worse as dfm). The following numbers for M10 are valid for my samples only. With 9.5dB, fm-decoding gives approx 40 OK-frames, didn't count bit errors yet. However, --iq2-demodulation improved decoding without much optimization effort. At 9.5dB it gives 100 OK frames, at 9.0dB it gives 72 OK. (8.5dB: 64 OK, 8.0dB: 20 OK, from here error correction would be fine...) csdr: ... | csdr fir_decimate_cc 2 0.005 | csdr bandpass_fir_fft_cc -0.21 0.21 0.01 | ... DFM also showed this kind of improvement. Both M10 and DFM don't have much BT, so simple FSK-IQ-decoding should work better anyway, rs41/rs92 have more pulse shaping/ISI. And somehow the FM-decoding of DFM and M10 seems not so nice, both amplitude and pulse shaping could be a reason, or maybe I should try more real world signals.

(for DFM and M10, the dB/noise values are with respect to symbol rate, DFM:2500, M10:9600)

darksidelemm commented 5 years ago

Closing this as I've added in the fsk_demod experimental chains as options in https://github.com/projecthorus/radiosonde_auto_rx/releases/tag/v1.1.0

@rs1729 - I still need to look into replacing the M10 and DFM demodulators with those from demod/mod. It looks like the only difference with dfm09mod is that you're not making assumptions about the ID code of the DFM-17/PS15 sondes, which is probably a good move until we have physical samples and can confirm what is what... Will probably switch to that in the next minor update. As for the M10 demod, how does your ID generation compare to what @Viproz does? I'll look at adding the dxlAPRS-compatible callsign generation to m10mod and submitting a PR.

rs1729 commented 5 years ago

M10: My ID is the actual M10 serial number, only with "M10-" prefix. It should be the same as Viproz's, only he added "T-" (for Trimble) or "G-" (gtop). M10-310-2-11600 instead M10-T-310-2-11600 (In the tracker-website it is a long ID. My "G" sonde startet with 2xxxx in the last part.) For gtop-data I have a separate decoder, and I think the M10 with gtop-GPS were only for a short period. Now they have the new Airprime XM1110, which is the successor of the gtop FireFly, and they output a Trimble-compatible data set again. Probably they have a custum GPS sentence, then in the firmware they produce the old Trimble data set. Thus there is now (real) difference. Only that the old M10 with Trimble Copernicus has the WNRO-issue. Last week there were M10 from 2015 flying, date 1999 ..., the WNRO-fix would help, if the week is before 2005 (approx trimble-copernicus release), it adds 1024.

darksidelemm commented 5 years ago

Yep, I have a sonde on my shelf that reports the 1999 date... I agree with removing the T- and G- additions. I'll look into switching back to m10mod if I don't hear anything from @Viproz

rs1729 commented 5 years ago

For the WNRO-fix I used a rather old approx "firmware"-date, so their should not be problems when processing old recordings. Don't know if a manual setting is needed (c.f. rs92), the 2005-date should be good for the next 5-6 years.