fft calculated using iq_server looks a bit strange

[flux242]

Hello,

I'm calculating fft using csdr and using iq_server. So, I suppose that the result should look the same for the same signal but there are some slight differences which makes me wonder if something isn't quite right.

Here is the picture of the spectrum I get using csdr https://i.imgur.com/G9A6erQ.png Here is the picture of the same region I get using iq_server/iq_client https://i.imgur.com/tazPe0s.png

The pictures are almost the same except that bin in the middle that goes down in case of iq_server. Additionally, the signal strength of the signal on the left (the bigger peak between 402.5 and 403 MHz) is somewhat weaker in case of iq_server

The sample rate is 2400000, number of bins 4096, gain 40, averaging 5 seconds (yes, I patched the iq_server a bit) (Different noise levels do not play any role)

Thanks

[flux242]

is this DC removal?

[rs1729]

Oh, yes. Just checked a few samples and was wondering why I put it there in the first place. I had this dc-removal in another fft-scan function, but it is not so accurate. And then I added dc-removal when reading the samples, because it is needed for better decoding (converting bits/sample can already add dc). So I should remove it when doing the FFT. Thanks.

[flux242]

so, I checked again and here how it is look like https://i.imgur.com/p6rGHMH.png I still see that the signal goes through a dc removal because the dc spike is missing on the right. The dc removal also reduces the signal power drastically. I'm not sure if this is the desired effect

[rs1729]

Yes, there is still a dc removal. (I only removed the second removal that I forgot to remove.) I want to remove dc because it can disturb decoding.

Can you give a little more information on the FFTs and the signals in your picture? The left is csdr, and on the right side, is this the inital server-side fft?

The dc that is removed here is the average over essentially the baseband sample rate, i.e. 1 second. (The length is building up at the beginning, so the inital server-side --fft might be a little different than client-side fft later, but the signal would change anyway, difficult to copmpare.) One could also try other windows and/or a moving average, but 1 second should capture a running radiosonde signal. Then the FFT is also the average over 2 sec (if I remember correctly). So all together it's not just bin 0. Thus if these signals are e.g. RS41 that transmit only half a second, you would see a lower signal power in the FFT if you average the FFT over 1 second or more. More so for M10. Thus RS41 signals appear weaker in the long-average FFT. If the baseband is just the narrow IF signal, then the "inner" average for dc removal might not be exactly what you want, too much signal in the baseband.

I don't know how csdr does the FFT, the are probably different choices you can make. I would not want a too nervous FFT with all the spikes, though not too smooth either.

Here I have added an option to switch off dc removal with "--dc0" for iq_server https://github.com/rs1729/RS/tree/test_iq_svcl_dc/demod/iq_svcl so you can compare e.g. ./iq_server --fft fft_dc1.txt iq_data.wav ./iq_server --dc0 --fft fft_dc0.txt iq_data.wav I didn't see much of a difference for the signals, only the dc at frequency 0.

[flux242]

I'll give a bit more info on the weekend and also will test without dc...

[rs1729]

If you compare the signals, consider the FFT and how it is integrated to get an average over 1-2 seconds to get a more static view on the spectrum.

The average or dc component over 1 sec (in time domain) would be bin 0 if the FFT size would be the sample rate (up to a normalization factor N=number_samples depending on how you define the DFT). Often the FFT size is much smaller depending on the resolution you want, and updated more frequently (if you are interested in a dynamic picture). Here the FFT is integrated at least 1 second, because the main reason for FFT here is to detect radiosonde signal peaks. (One could also take longer FFTs and condense the bins to the frequency resolution that is needed.) So that's a different kind of average for all bins in frequency domain. Now there are radiosondes that don't transmit continuously, so the integrated spectrum would show e.g. RS41 signals weaker than they are. But this is not a big problem, when scanning for peaks of a certain bandwidth and shape (symmetric), the detection should be sensitive enough to find potential radiosonde signals. (Only the M10 has this carrier at the upper frequency, that's not so nice, and the transmissions are rather short.) One could probably take more slices and select only the parts where a signal is above the noise, but I don't think it's worth the effort here. So if the signals in your picture are RS41, then it shouldn't be surprising that they appear to be weaker than in a live spectrum when you look at it when the signal is on. However I don't know how you did the FFT with csdr and how it can be compared the integrated (summed) FFT of iq_server.

[flux242]

so, there are 2 rs41 sondes on the picture I posted above. But, I do averaging over 5 seconds both with csdr and iq_server. Number of bins is also the same 4096. Number of FFT's per second is the same - 20 (i.e 100 scans are averaged in 5 sec). I tested the --dc0 option and still I see the difference in the power level. I don't understand why but to my opinion the result should be the same. You may check the https://github.com/flux242/radiosonde/blob/master/scripts/receivemultisonde.sh script to see how the power scanning is implemented using csdr and iq_client in scan_power and scan_power2 functions. The difference is that the scan_power and csdr receive the IQ samples directly and scan_power2 simply requests the results from the iq_server.

[flux242]

so, there are 2 rs41 sondes on the picture I posted above. But, I do averaging over 5 seconds both with csdr and iq_server. Number of bins is also the same 4096. Number of FFT's per second is the same - 20 (i.e 100 scans are averaged in 5 sec). I tested the --dc0 option and still I see the difference in the power level. I don't understand why but to my opinion the result should be the same. You may check the https://github.com/flux242/radiosonde/blob/master/scripts/receivemultisonde.sh script to see how the power scanning is implemented using csdr and iq_client in scan_power and scan_power2 functions. The difference is that the scan_power and csdr receive the IQ samples directly and scan_power2 simply requests the results from the iq_server.

[rs1729]

Don't know what the output of the csdr fft is, but I see what's very likely the difference (-> "logaverage" because of dB). In iq_server I just sum the FFTs, though not the FFT values, but the dB, so I shouldn't call it the average or integrated FFT! And if I think about it, it is not only faster if the db-log is not calculated every FFT-round, it is also closer to reality. The stronger signals don't lose that much if they don't transmit continuously like RS41 and M10. This average would be more like taking the long FFT and condensing the bins. For comparison/testing I added it as option --avg_db in test_iq_svcl_dc branch, but this should be the default I guess, since the sum of dBs is more expensive to compute and it has no real meaning, so it is not really needed.

[flux242]

with the --avg_db the problem is solved. Thanks

[rs1729]

Ok, I made the avg dB the default. More meaningful and probably faster. (In test_iq_svcl_dc --sum_db is now the old sum, but probably I will delete this testing branch eventually.) Don't know if you really need the no-dc-removal option? Thanks for testing and pointing this point.

[flux242]

no, I don't need the no-dc-removal option and it'll work for me equally good anyway because I'm adding a hi-pass filter at 10 Hz using sox command (it's in the script I linked above - start_decoder function). But maybe this option could be useful in some use cases for somebody else? I'm not sure