search

jks-prv / Beagle_SDR_GPS

KiwiSDR: BeagleBone web-accessible shortwave receiver and software-defined GPS
http://kiwisdr.com
470 stars 158 forks source link

IQ AGC distortion #220

Closed DazDSP closed 5 years ago

DazDSP commented 5 years ago

In IQ mode, AM carriers are often very low frequency - less than a few Hz. This is because they are near zero-beat. They can even be DC!

The carriers (and thus the sidebands as well) are distorted by AGC action that flattens the peaks. See image. AGCclipped

This appears to show the current KiwiSDR AGC approach is using simple peak detection, which is not ideal.

This gain modulation ripple of the AGC causes harmonics of the carrier, and causes IMD of the sidebands, degrading the signal.

This AGC ripple may also degrade other signals that cross over the 0Hz region of the baseband - such as DRM.

Since the signal is in IQ format, the AGC should be using "Vectorsum" or "Magnitude Estimation" to produce an almost ripple free AGC signal on sinewaves. This can then feed into the normal AGC time constants.

The Magnitude Estimation method is shown here: https://dspguru.com/dsp/tricks/magnitude-estimator/

jks-prv commented 5 years ago

I just used the AGC module from CuteSDR. I'm not even sure how it works exactly. Feel free to submit a patch..

DazDSP commented 5 years ago

From the CuteSDR documentation: IQenv

I think we found what the significant difference is!!! Obviously they didn't want to compute a square root...

That's why I posted about the "Magnitude Estimator" DSP trick (no square root needed). It works very well and is fast, though the accuracy can be as bad as 6%. Probably good enough for AGC use though. With a little extra code, it's accuracy can be improved to better than 1%.

In the CuteSDR quote above, the first method - Vectorsum - is best. The second method is essentially a peak detector. That's the cause of the distortion shown in the IQ waveform image above. It's very hard to eliminate that with AGC filtering, because with IQ signals that cross 0Hz there are some extremely low frequencies present.

The log function isn't necessary - though maybe it's included within the attack/decay time constant processing that I've been using in Faust... In Assembler I have been using linear, slew limited attack and decay.

Forward acting AGC is quite simple, and unconditionally stable (unlike feedback loop AGC systems). These are the steps:

Compensate for delays in the smoothing attack by delaying the IQ signal by a similar amount before performing that multiplication to prevent overshoots.

Hang can be added by setting a timer whenever the envelope signal increases above the current smoothed value (the attack phase). The current value is not decayed until the timer expires. Then it is decayed back towards zero as usual.

Using a slope in the AGC (a ratio of less than infinity to one) to reduce the severity of it's action can sound better (less harsh), but adequate headroom is needed or the signal will clip or overflow causing severe distortion.

Unfortunately, C isn't my language... I program mainly in Assembler and Faust.

Maybe I could write some Faust code and export it to C source, but someone who knows C would need to convert that into a patch.

Daz

DazDSP commented 5 years ago

Fixing this issue may be as simple as enabling the code on line 196 in agc.cpp. Is that the Vectorsum method? Yes, it is!

So change line 189 of agc.cpp from:

if 1

to:

if 0

And this issue should be fixed!

jks-prv commented 5 years ago

What sort of settings and signals are you using to get the AGC clipping your first post shows? I've tried IQ mode with various signals and threshold settings of the AGC and all the resulting IQ files produced by kiwirecorder look fine -- no clipping. Before I go changing stuff for release I need a way to verify that a problem is actually being fixed.

The alternative code from the "#if 1" doesn't have a sqrt(). In fact it seems to be using a magnitude estimation = 0.5 * power(I,Q), where power = vectorsum(I,Q). The current code just appears to be saving a couple of multiplies and a divide over the alternative.

jks-prv commented 5 years ago

The Kiwi built-in record function (as opposed to recording with the external kiwirecorder Python script) seems to have problems in IQ mode. The "left" channel is full scale noise and the sample rate appears to be wrong (e.g. recording a CW station will playback with a lower sidetone). Files recorded with all the other modes seem fine.

Are you plotting IQ data using audio directly via a VAC as opposed to recording to a file first?

DazDSP commented 5 years ago

It isn't really AGC clipping because the modulation is not removed from the peaks. It is AGC ripple that appears to be caused by using the peak detector method.

There is no way to avoid this AGC distortion on low frequency signals (say under 1Hz) when a peak detector is used because the peak detector is only producing an output on signal peaks, and the AGC filtering/sliding window is too fast to smooth them out.

This is often the case on AM signals because the carrier is very close to zero beat. It may also be worse on fading signals that are working the AGC harder.

I have tried setting the AGC recovery time to the maximum 5000mS but it didn't help. It's also getting too slow to be useful on fading signals.

The Vectorsum method produces a DC output proportional to the sinewave envelope. On a stable sinewave there is no significant ripple. This should remove this source of AGC ripple.

I use IQ mode for AM Sync ISB reception of shortwave signals twice a week, and I often see the AGC distortion on carriers. Sometimes it's also audible on the modulation.

The built-in recording of IQ data works very well, and the end result is very good despite the AGC effects. But it could be better.

Make sure your sound editor can handle the particular file format that is saved, or it will cause problems. Audacity works fine.

My IQ image was an IQ recording made in the browser and opened in Audacity. Normally I run the volume about 3/4 or slightly less. That time I reduced the volume to halfway to drop the waveform height to ensure nothing else could be clipping.

Ideally, it shouldn't be possible to cause distortion on a signal unless AGC is turned off completely and the gain is raised until it overloads the receiver.

Apparently it is possible to find a square root by using logarithms.. The agc.cpp comment says "//mag is power so 0.5 factor takes square root of power".

To see the AGC problem for yourself:

Here's an example at the moment: http://50.251.11.145:8073/?f=12160.00iqz14

jks-prv commented 5 years ago

Okay, I guess I believe you now (thanks for the detailed instructions on how to reproduce the problem). Below is a screenshot (click for larger) of browser IQ recordings without (red arrow) and with (green arrow) the alternative code from the "#if 1". I used a fading shortwave signal and 10 Hz bandwidth. Roughly 1 Hz carrier offset. The fuzzy patches are static crashes.

I'll include this fix in today's update. It would be great if this results in improved DRM reception. Thank you.

image

jks-prv commented 5 years ago

Fixed in v1.276 release.