Slow gain changes (> 3 dB!) on Booster 003B002E3037470535353239

[dnadlinger]

Observation

I am seeing the gain of one channel change by >3 dB on a TechnoSystem v1.4 Booster (and potentially also a few other Creotech/TechnoSystem devices, although without good data).

The Booster in question is usually driven by an Urukul and drives an AOM. Measuring the powers directly with an RF power meter at two points, when the gain appears high/low, gave:

• "High gain": -11.45 dBm input at 210 MHz gives 28.29 dBm output -> 39.74 dB gain
• "Low gain": -11.48 dBm input at 210 MHz gives 24.42 dBm output -> 35.90 dB gain

There was no clear short-term trend visible, but the gain had changed back and forth between roughly these values over a few days.

Debugging information

The problem occurs on channel #0 on Booster 003B002E3037470535353239, which is a factory-refurbished older release.

No I2C errors, nothing interesting in the log:

> i2cerr
        #0  #1  #2  #3  #4  #5  #6  #7
I2C ERR     0   0   0   0   0   0   0   0

> logstash
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[ERROR] Interlock tripped on channel 1, i=0 o=1

(Channel 1 is unrelated.)

Status without RF applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.00    0.13    0.00    0.08    0.02    0.01    0.00    0.05
TXPWR [V]   0.01    0.01    0.15    0.12    0.01    0.01    0.01    0.02
RFLPWR [V]  0.04    0.01    0.01    0.01    0.01    0.22    0.13    0.08
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 5.00    5.00    5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    -4.31   -4.48   -4.24   -3.55   -2.58   -2.93   -0.01   -4.04
I30V [A]    0.028   0.042   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.246   0.255   0.245   0.249   0.250   0.249   0.253   0.256
5V0MP [V]   4.934   4.930   4.940   4.950   4.920   4.942   4.940   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        15  19  244 195 13  13  13  38
ADC2        63  18  15  13  13  364 205 123
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Status with above RF power applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.80    0.89    0.00    0.08    0.03    0.01    0.00    0.05
TXPWR [V]   1.57    1.81    > 0.15  0.12    0.01    0.01    0.01    0.03
RFLPWR [V]  0.64    1.06    0.01    0.01    0.01    0.22    0.13    0.07
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 25.41   29.72   5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    8.01    16.78   -4.24   -3.54   -2.58   -2.95   -0.01   -4.05
I30V [A]    0.090   0.162   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.244   0.254   0.245   0.249   0.250   0.249   0.253   0.257
5V0MP [V]   4.930   4.928   4.940   4.950   4.918   4.942   4.942   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        2578    2972    245 196 14  14  13  42
ADC2        1051    1741    15  14  14  366 213 121
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Note the low bias current.

Disabling and re-enabling the channel using the console (disable 1; enable 1) doesn't change much about the gain.

Bias currents after successively disabling and re-enabling channel 0: 32 mA, 37 mA ,41 mA, 43 mA, still 43 mA, 44 mA, 45 mA. Under load, still only 88 mA and 25.25 dBm forward power, though (using internal detector, but seems to roughly match power meter).

(Lack of) gain is stable across power-cycling entire Booster. Initially had 36 mA current after startup, settled to 41 mA after a few minutes.

Channel 0 supply current vs. input RF amplitude (in Urukul full-scale units, where 1.0 corresponds to the above measurements):

DDS amplitude / FS	I30V / A
0.0	0.032
0.1	0.038
0.2	0.042
0.3	0.049
0.4	0.057
0.5	0.059
0.6	0.064
0.7	0.069
0.8	0.074
0.9	0.081
1.0	0.089

Extra observations

While attempting to gather data on this Urukul -> Booster -> AOM channel behaviour a few days ago, I saw the gain initially starting out low when switching on the Urukul channel. About 0.5 s later, the output power would recover to the high value in what looked very much like a step change. (This was done while acquiring AOM optical output power readings, i.e. by proxy RF power input measurements, for points between zero and full RF amplitude in random order. The gain really did appear to change for all input powers.) I couldn't conclusively establish this as being caused by Booster, as the problem went away after some 10 minutes before I could take a closer look.

[dnadlinger]

(I'm not sure what is going on here. Some intermittent almost-short? Weird damage to the power FET or biasing circuitry? Other RF black magic seems unlikely, as the bias current is low, not high, and it occurs without input.)

[hartytp]

Thanks for the report.

I plan to post this booster back to creotech tomorrow for investigation (@dnadlinger does it also have watchdog issues?)

[hartytp]

We dismissed these problems as damaged FETs before, but I don’t see how the FETs could become damaged when there is fold back limiting.

Can’t be rf oscillations as that would push the bias current up normally not down

Can’t be disconnected bias voltage as that would push the current up

The fold back protection hasn’t tripped as the current can still increase with input rf power

Quite mysterious...

[gkasprow]

I hope to be able to recreate it in my lab. This is really mistery. I will connect the channel outside of the booster to exclude the influence of the other circuits.

[hartytp]

Yes, I’ll definitely owe you a beer if you can figure this one out!

[hartytp]

FYI I if you look at they hysteresis issue you’ll see that I found that when I power cycled the booster the 28V voltage and gate bias current were reproducible to better than 1 mag. However the bias current (measured directly across the sense resistor ) could change by 10% or more

[gkasprow]

We used to have some hysteresis a long time ago when the power stage output indictor had too low value. I could not explain why this happened, but since higher value solved the issue, we didn't dig any deeper.

[gkasprow]

Could be also an issue with the output capacitor. Some ceramic caps behave in a bizarre way when high current is passed at higher frequencies. Changing to bigger package size helps. I already experienced such a situation in my amplifier working with 10W and 400MHz a few weeks ago.

[hartytp]

Yes but that doesn’t explain this since it happens at dc with no rf signal...

[dnadlinger]

Also, the ("good") output power I'm using is only ~ 29 dBm, so if it was just related to raw power, surely it would routinely occur on channels tested on full blast?

[hartytp]

@gkasprow we're shipping two Boosters to Creotech today.

One has a missing channel (the one with a short we sent back). At some point the watchdog tripped on this booster and now it won't start up properly even after power cycling.

The other one has all channels populated, and is the one with the gain variations. I believe that both should manifest the watchdog issues.

[gkasprow]

I managed to recreate the problem with one channel you sent us previously. This channel shows rapid gain changes. It does it from time to time, mostly during a few minutes after power on. Later on, it seems to work correctly. I'm observing the signal at the power stage entry and it is stable. I plan to look at the gate DC voltage and current using same scope.

[hartytp]

FWIW it's not clear that this is really an issue with the power stage. Could be anywhere in the signal chain. So the first thing to do is probably to trace the signal through Booster and see where the issue really is.

[gkasprow]

As I wrote, signal at the gate looks OK, the same with its DC level. I will look at the current

[hartytp]

aah, ok, sorry I mis-understood you. So...it's definitely the power FET/matching then.

[gkasprow]

it looks so. However it seems that after heating up it works.

What is the issue with the channel you posted me a few days ago?

[hartytp]

What is the issue with the channel you posted me a few days ago?

https://github.com/sinara-hw/Booster/issues/332

[hartytp]

@gkasprow which channel were you looking at? Did you read @dnadlinger's post at the top of this issue? In channel 0 (ID 02:22) the gain is unstable and slowly varies between high and low values. There is a corresponding change in the bias current.

The bias current also shows significant hysteresis between power cycles (but, interestingly not between enabling/disabling the channels). I've observed this behaviour on other channels, but not as bad.

[hartytp]

The bias current for this channel is generally lower than the 50mA it was calibrated for.

[hartytp]

However it seems that after heating up it works.

We observed the gain issues to come and go.

[dnadlinger]

However it seems that after heating up it works.

[…]

We observed the gain issues to come and go.

Just to reiterate, we definitely observed large gain changes with the module, i.e. the entire Booster, constantly powered on.

(This is talking about bias power, though; the RF signal on that channel does vary as part of the experiments in the ion trap, between completely off, different power levels, and a few tens of MHz of frequency difference.)

[gkasprow]

the issue is more complex the output transistor seems to ge internal avalanche discharge. Look at the plots:

[gkasprow]

interesting plot

[gkasprow]

the voltage on gate looks very distorted. And the power stage seems to amplify correctly.

[gkasprow]

It looks like the first stage got crazy

[hartytp]

Aah amazing! So we’re closing in on the real issue now...

[gkasprow]

I replaced the amp and now it works better. No idea why it behave like this. It could be damaged by excessive input power.

[hartytp]

Hmm...

Which amp did you replace? How can it be damaged when we have the interlock?

I think the input stage playing up could be the cause of many issues (intermittent noise/hysteresis on the 29V bias current, strange variations etc) which I have seen on many channels including some that are fresh from the supplier. So I think there is a systematic issue here

[hartytp]

We need to get to the bottom of this and not just brush it under the carpet

[gkasprow]

I replaced the first stage only. it is supplied from 5V. It could be damaged during experimental work. I don't know history of this channel. However, after replacement, I've noticed that the gain drifts slowly after power-up. So there is still something happening here. I cannot re-create it anymore. Could be also cool-down effect after amp soldering but I doubt.

[gkasprow]

I cycled the power and the gain is stable. But I'm pretty sure I saw a slow drift by something like 3dB...

[gkasprow]

I will continue tomorrow. It's time to go home.

[gkasprow]

I left it running with full power overnight

[gkasprow]

I'm thinking about the possible causes of ADL5536 amplifier damage. It is loaded with a high pass filter which has a very low capacitance value of 39pF. On the other had it has 1uH load. What if we overdrive it? The output stage is just a common emitter transistor that switches between saturation and off state. And what happens with inductor energy when the transistor switches to off state? It generates a high voltage that damages the transistor. Its energy is not that high to damage it thermally, it just causes avalanche discharge and degradation of the silicon structure. Such a situation could happen here. Solution could be simple - just add low capacitance TVS (or two silicon diodes in series) or move the highpass filter after the attenuator. I will try to recreate such a condition tomorrow. Here is the simulation I did. I used a popular RF transistor and a similar LC circuit.

[gkasprow]

the reference design uses 100nF cap and 50Ohm load of next stage so we get much lower overshoot

[gkasprow]

We can cause such conditions for example when activating the input interlock or during disconnecting or connecting the input cable. Anyway, I will try to damage the amplifier I replaced in such a way tomorrow observing the output with a low capacitance active probe.

[gkasprow]

The output stage has a max voltage of 100V so it gives us safe margin.

[gkasprow]

The second stage

[hartytp]

@gkasprow what are you measuring in these scope traces?

[gkasprow]

Green is amplifier output via 20dB attenuator Yellow is power stage gate

[hartytp]

With RF input applied? Or, are you saying the amplifier was free oscillating at around 400MHz with input terminated? RMS around 1.2V so about 14dBm?

[hartytp]

Also, is the power stage really amplifying correctly? The green curve doesn't look like just a scaled up version of the yellow curve.

[gkasprow]

I used a 400MHz generator all the time. -10dBm The amplifier was itself switching between some internal states :)

[gkasprow]

It was caused by broken first stage which sometimes was amplifying, sometimes was terribly distorting the input signal.

[hartytp]

I see. So the observation is that the gate voltage is distorted and the output switches between different gains as previously observed. OK.

[gkasprow]

not really. The output simply follows the gate voltage. But I observe the output after the HP filter so it does not look that bad as the input.

[hartytp]

Without any signal applied did you see any AC at the PA FET's gate/output?

[gkasprow]

nope.

[gkasprow]

It looks like the gain changes were caused by imput amplifier. I replaced it with one disassembled from broken channel. So it can be broken as well.

[hartytp]

Okay. I suspect that there is more than one issue here. The symptoms we have observed are:

1. gain changes
2. sometimes the gain changes are accompanied by a change in the PA FET DC bias (i.e. the current is lower even with no RF applied)
3. Hysteresis in the FET bias current when the unit is turned on/off (going in/out of standby doesn't affect this. It's also not thermal since the current seems stable until the unit is power cycled again)
4. Intermittent periods when the 29V FET bias current becomes nosier

(1) could be explained by damaged first stage amps (2) can't be explained by damaged first stage amps (3)-(4) could be explained if the first stage amps were oscillating or something like that.

Anyway, if the first stage amp is damaged we should definitely look at that first, but it may be that there is more than one issue here...