Slow gain changes (> 3 dB!) on Booster 003B002E3037470535353239

[dnadlinger]

Observation

I am seeing the gain of one channel change by >3 dB on a TechnoSystem v1.4 Booster (and potentially also a few other Creotech/TechnoSystem devices, although without good data).

The Booster in question is usually driven by an Urukul and drives an AOM. Measuring the powers directly with an RF power meter at two points, when the gain appears high/low, gave:

• "High gain": -11.45 dBm input at 210 MHz gives 28.29 dBm output -> 39.74 dB gain
• "Low gain": -11.48 dBm input at 210 MHz gives 24.42 dBm output -> 35.90 dB gain

There was no clear short-term trend visible, but the gain had changed back and forth between roughly these values over a few days.

Debugging information

The problem occurs on channel #0 on Booster 003B002E3037470535353239, which is a factory-refurbished older release.

No I2C errors, nothing interesting in the log:

> i2cerr
        #0  #1  #2  #3  #4  #5  #6  #7
I2C ERR     0   0   0   0   0   0   0   0

> logstash
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[ERROR] Interlock tripped on channel 1, i=0 o=1

(Channel 1 is unrelated.)

Status without RF applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.00    0.13    0.00    0.08    0.02    0.01    0.00    0.05
TXPWR [V]   0.01    0.01    0.15    0.12    0.01    0.01    0.01    0.02
RFLPWR [V]  0.04    0.01    0.01    0.01    0.01    0.22    0.13    0.08
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 5.00    5.00    5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    -4.31   -4.48   -4.24   -3.55   -2.58   -2.93   -0.01   -4.04
I30V [A]    0.028   0.042   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.246   0.255   0.245   0.249   0.250   0.249   0.253   0.256
5V0MP [V]   4.934   4.930   4.940   4.950   4.920   4.942   4.940   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        15  19  244 195 13  13  13  38
ADC2        63  18  15  13  13  364 205 123
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Status with above RF power applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.80    0.89    0.00    0.08    0.03    0.01    0.00    0.05
TXPWR [V]   1.57    1.81    > 0.15  0.12    0.01    0.01    0.01    0.03
RFLPWR [V]  0.64    1.06    0.01    0.01    0.01    0.22    0.13    0.07
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 25.41   29.72   5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    8.01    16.78   -4.24   -3.54   -2.58   -2.95   -0.01   -4.05
I30V [A]    0.090   0.162   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.244   0.254   0.245   0.249   0.250   0.249   0.253   0.257
5V0MP [V]   4.930   4.928   4.940   4.950   4.918   4.942   4.942   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        2578    2972    245 196 14  14  13  42
ADC2        1051    1741    15  14  14  366 213 121
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Note the low bias current.

Disabling and re-enabling the channel using the console (disable 1; enable 1) doesn't change much about the gain.

Bias currents after successively disabling and re-enabling channel 0: 32 mA, 37 mA ,41 mA, 43 mA, still 43 mA, 44 mA, 45 mA. Under load, still only 88 mA and 25.25 dBm forward power, though (using internal detector, but seems to roughly match power meter).

(Lack of) gain is stable across power-cycling entire Booster. Initially had 36 mA current after startup, settled to 41 mA after a few minutes.

Channel 0 supply current vs. input RF amplitude (in Urukul full-scale units, where 1.0 corresponds to the above measurements):

DDS amplitude / FS	I30V / A
0.0	0.032
0.1	0.038
0.2	0.042
0.3	0.049
0.4	0.057
0.5	0.059
0.6	0.064
0.7	0.069
0.8	0.074
0.9	0.081
1.0	0.089

Extra observations

While attempting to gather data on this Urukul -> Booster -> AOM channel behaviour a few days ago, I saw the gain initially starting out low when switching on the Urukul channel. About 0.5 s later, the output power would recover to the high value in what looked very much like a step change. (This was done while acquiring AOM optical output power readings, i.e. by proxy RF power input measurements, for points between zero and full RF amplitude in random order. The gain really did appear to change for all input powers.) I couldn't conclusively establish this as being caused by Booster, as the problem went away after some 10 minutes before I could take a closer look.

[hartytp]

NB lower gains correspond to lower DC bias currents. That suggests there is something wrong in the FET/biassing circuit. However, last time I looked the bias voltage on the supplemental control voltage wasn't changing.

[hartytp]

@gkasprow how do you know it was the ADL and not the PHA? Did you trace the signal through the circuit and see where the distortion came in?

[gkasprow]

yes. I used <1pF oscilloscope probe. To access the lower board I have a right angled goldin adapter and pair of DSUB connectors connected with 10cm wires.

[hartytp]

yes.

"Yes", as in you traced the signal through and confirmed that the first amp was causing the distortion?

If so, that's great! We've caught one place where there is damage so we can fix that...

[gkasprow]

exactly.

[hartytp]

NB the ADL can take up to 20dBm input power. Given that there is approx 7dB of loss before the amp it should take nearly 0.5W of input power to damage it. The only thing we've used to drive Booster is Urukul so I think we can safely say that input overdrive is not the issue here!

[gkasprow]

Yes, but it can lead to output overvoltage.

[hartytp]

I don't see a spec on max collector voltage.

[gkasprow]

it's probably the same as twice the supply voltage

[hartytp]

Is that an RF rule of thumb? So the worry would be some kind of LC resonance in the output stage? Maybe coupled with a switch glitch either from Urukul or from the interlock?

[gkasprow]

I assume the sine wave and AC coupled load. Output can go to zero and twice the supply. The max power supply is then transistor breakdown value /2. I can measure the breakdown voltage tomorrow.

[hartytp]

I guess the other worry is if the RF is somehow applied before the amp is properly biassed?

[gkasprow]

without biasing there is no overvoltage induced.

[hartytp]

No, but isn't there a possible issue of damaging the input stage with RF applied before bias?

[dnadlinger]

@hartytp: I guess at abs max. V_cc you'd assume (without further information) that a symmetric signal from 0 to 2 V_cc is the maximum permissible. I was too slow, apparently. :) Regarding input damage, at the input powers we typically use with Urukul, we can't even forward-bias a Schottky junction, so seems unlikely… Interlock trip transients seem more promising.

@gkasprow: Thanks a lot for concentrating on these issues now – really looking forward to when Booster just works!

[hartytp]

Okay, so anything above ~12V could cause breakdown?

[gkasprow]

the IC is just Darlington transistor with matched input

[gkasprow]

exactly

[hartytp]

Okay. That makes sense. Could be what is killing the input stage. As you say, it's worth measuring the spikes we get when the interlock trips (maybe also hook up to Urukul and look at those transients) and see how it compares.

[gkasprow]

RF amplifiers usually have a similar schematic

[hartytp]

In an earlier version the input attenuator was after the initial pre-amp stage.

We moved it to give a better input return loss, but that's really not necessary IMHO. If there is an issue, we could always move the input attenuator to after the pre-amp. That would isolate it from the load.

[gkasprow]

Tomorrow I plan:

• measure amplifier breakdown voltage
• measure spikes it produces during interlock activation or applying square wave signal
• looking deeper to slow gain changes that I observed today
• looking deeper into the hysteresis issue.

[hartytp]

Wonderful! Thanks!

[hartytp]

@gkasprow I think your plan for tomorrow sounds good.

AFAICT we've got damage to both the power FET and the first pre-amp in different boards. So, we should check the whole design by looking for voltage/bias current transients during: turn on/off; interlock trip/reset; standby/enable; Urukul switch transients? Basically check the whole thing over for any transient issue that can take one of the amp ICs outside its abs max ratings.

[gkasprow]

The channel I left working with full power overnight died. The first stage does not pass the signal anymore. Yesterday I assembled used chip which had some issues.

[gkasprow]

The amplifier died in a strange way - the input bias is 0

[hartytp]

the pre-amp input bias is 0V???

[gkasprow]

yes

[gkasprow]

but as I said, the chip was taken from a broken amplifier.

[hartytp]

what do the diagnostics (e.g. VCP start) say?

Anyway, at least we've identified on issue: the pre-amps are failing. That's a good start.

[gkasprow]

It's connected to slot 4

[gkasprow]

Another issue is too low current of the power stage

[hartytp]

Okay, so in this failure mode the 5V0 bias current is noticably lower than normal. I don't believe I've ever observed that, so it's a new failure mode. Joy of joys :)

The low PA current is something I have observed before. I wonder if the answer here is that voltage spikes propagate through the system and trash the three amps at various levels, with the FETs being the most sensitive?

[gkasprow]

I did some simulation of the FET stage but did not manage to exceed 100V on its output. But Let's measure it.

[hartytp]

Do you think it's excessive gate voltage when a high-frequency spike is produced?

[hartytp]

Otherwise, I have no idea how they are still dying so often

[gkasprow]

ah, yes. this could be an issue.

[gkasprow]

we can easily exceed max ratings for the power stage

[hartytp]

Well, the right approach here is to verify that we exceed that and ideally see a change in FET performance once we do. If that happens we can be decently confident we've identified the issue. (I had checked that max rating, but thought we were comfortably inside it because I hadn't appreciated how bad the ringing on the output stage can be...)

[gkasprow]

Broken gate has increased leakage current

[hartytp]

Broken gate has increased leakage current

How would the gate become damaged? have you observed a voltage spike there? Have you measured an increased leakage current?

Last time I checked, the bias voltage measured on the supplemental PCB seemed very stable. Admittedly that's only got a 4R7 impedance, while there is another 39R on the PA PCB. But if the gate leakage were changing a lot I would have expected to see changes in OUT_BIAS measured on the supplemental PCB. Unless something odd is happening like the feedback coupling large output spikes onto the input and damaging it that way?

[hartytp]

@gkasprow any more progress on this?

[gkasprow]

Gate can be damaged with too high voltage. It causes leakage which can be measured using ohmmeter. Yuo may not see it with current reading

[hartytp]

Okay, can you check for this.

[gkasprow]

I will. I have unexpected paperwork today.

[hartytp]

Understood. Hopefully we can finish the testing this week. I'm keen to get some repaired/patched units asap so we can start gathering statistics and see if all issues are finally fixed.

[gkasprow]

I'm back in the lab

[hartytp]

:)

[gkasprow]

I took new non-damaged unit. I was looking at the first stage output with 3.9pF scope probe. I applied 1V 1kHz square signal to the input. The positive pulses at the stage output do not exceed 10V (5V above 5V supply)

[gkasprow]

however this unit has other L2 model. It looks like FB, not wire-wound like the one in faulty channel. I will replace the ADL chip with another one in the PCB that caused troubles.