Slow gain changes (> 3 dB!) on Booster 003B002E3037470535353239

[dnadlinger]

Observation

I am seeing the gain of one channel change by >3 dB on a TechnoSystem v1.4 Booster (and potentially also a few other Creotech/TechnoSystem devices, although without good data).

The Booster in question is usually driven by an Urukul and drives an AOM. Measuring the powers directly with an RF power meter at two points, when the gain appears high/low, gave:

• "High gain": -11.45 dBm input at 210 MHz gives 28.29 dBm output -> 39.74 dB gain
• "Low gain": -11.48 dBm input at 210 MHz gives 24.42 dBm output -> 35.90 dB gain

There was no clear short-term trend visible, but the gain had changed back and forth between roughly these values over a few days.

Debugging information

The problem occurs on channel #0 on Booster 003B002E3037470535353239, which is a factory-refurbished older release.

No I2C errors, nothing interesting in the log:

> i2cerr
        #0  #1  #2  #3  #4  #5  #6  #7
I2C ERR     0   0   0   0   0   0   0   0

> logstash
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[INFO] network client disconnected
[INFO] network client disconnected
[INFO] network client 10.255.6.191 connected
[ERROR] Interlock tripped on channel 1, i=0 o=1

(Channel 1 is unrelated.)

Status without RF applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.00    0.13    0.00    0.08    0.02    0.01    0.00    0.05
TXPWR [V]   0.01    0.01    0.15    0.12    0.01    0.01    0.01    0.02
RFLPWR [V]  0.04    0.01    0.01    0.01    0.01    0.22    0.13    0.08
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 5.00    5.00    5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    -4.31   -4.48   -4.24   -3.55   -2.58   -2.93   -0.01   -4.04
I30V [A]    0.028   0.042   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.246   0.255   0.245   0.249   0.250   0.249   0.253   0.256
5V0MP [V]   4.934   4.930   4.940   4.950   4.920   4.942   4.940   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        15  19  244 195 13  13  13  38
ADC2        63  18  15  13  13  364 205 123
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Status with above RF power applied:

PGOOD: 1
FAN SPEED: 20 %
CHANNELS INFO
==============================================================================
        #0  #1  #2  #3  #4  #5  #6  #7
DETECTED    1   1   1   1   1   1   1   1
HWID        02:22   CD:F9   7B:9F   20:8D   E8:27   4F:79   21:12   C6:3A
INPWR [V]   0.80    0.89    0.00    0.08    0.03    0.01    0.00    0.05
TXPWR [V]   1.57    1.81    > 0.15  0.12    0.01    0.01    0.01    0.03
RFLPWR [V]  0.64    1.06    0.01    0.01    0.01    0.22    0.13    0.07
INPWR [dBm] nan nan nan nan nan nan nan nan
TXPWR [dBm] 25.41   29.72   5.00    5.00    5.00    5.00    5.00    5.00
RFLPWR [dBm]    8.01    16.78   -4.24   -3.54   -2.58   -2.95   -0.01   -4.05
I30V [A]    0.090   0.162   0.048   0.046   0.042   0.049   0.051   0.045
I5V0 [A]    0.244   0.254   0.245   0.249   0.250   0.249   0.253   0.257
5V0MP [V]   4.930   4.928   4.940   4.950   4.918   4.942   4.942   4.962
ON      1   1   1   1   1   1   1   1
SON     1   1   1   1   1   1   1   1
IINT        0   0   0   0   0   0   0   0
OINT        0   0   0   0   0   0   0   0
SINT        0   0   0   0   0   0   0   0
ADC1        2578    2972    245 196 14  14  13  42
ADC2        1051    1741    15  14  14  366 213 121
INTSET [dBm]    31.04   30.84   30.35   28.99   36.87   36.67   35.49   33.59
DAC1        4095    4095    4095    4095    4095    4095    4095    4095
DAC2        3413    3424    3527    3341    3992    3896    3924    3821
SCALE1      83  85  82  83  87  88  85  87
OFFSET1     470 446 727 619 460 375 565 571
BIASCAL     1865    1539    1879    1935    1527    1939    1761    1929
HWIS        82.08   84.33   83.17   82.83   85.17   83.92   83.00   85.25
HWIO        865.08  823.33  1003.17 939.83  852.17  818.92  978.00  957.25
LTEMP       27.50   28.25   28.50   27.00   28.50   27.00   27.00   28.00
RTEMP       27.50   27.00   27.00   28.00   27.00   28.25   28.25   27.00
==============================================================================

Note the low bias current.

Disabling and re-enabling the channel using the console (disable 1; enable 1) doesn't change much about the gain.

Bias currents after successively disabling and re-enabling channel 0: 32 mA, 37 mA ,41 mA, 43 mA, still 43 mA, 44 mA, 45 mA. Under load, still only 88 mA and 25.25 dBm forward power, though (using internal detector, but seems to roughly match power meter).

(Lack of) gain is stable across power-cycling entire Booster. Initially had 36 mA current after startup, settled to 41 mA after a few minutes.

Channel 0 supply current vs. input RF amplitude (in Urukul full-scale units, where 1.0 corresponds to the above measurements):

DDS amplitude / FS	I30V / A
0.0	0.032
0.1	0.038
0.2	0.042
0.3	0.049
0.4	0.057
0.5	0.059
0.6	0.064
0.7	0.069
0.8	0.074
0.9	0.081
1.0	0.089

Extra observations

While attempting to gather data on this Urukul -> Booster -> AOM channel behaviour a few days ago, I saw the gain initially starting out low when switching on the Urukul channel. About 0.5 s later, the output power would recover to the high value in what looked very much like a step change. (This was done while acquiring AOM optical output power readings, i.e. by proxy RF power input measurements, for points between zero and full RF amplitude in random order. The gain really did appear to change for all input powers.) I couldn't conclusively establish this as being caused by Booster, as the problem went away after some 10 minutes before I could take a closer look.

[hartytp]

how large are the switch transients?

[gkasprow]

these transients do not trigger interlock yellow - output of the first stage green - channel output after 20dB attenuator

[gkasprow]

interlock activation blue - 1st stage output yellow - input switch control signal

[gkasprow]

yellow - 1st stage output green - channel output I applied 1.5Vpp 10Hz square signal to the input

[gkasprow]

I looked at the power stage gate in same conditions green - PA output yellow - power stage gate rising edge and falling edge

[gkasprow]

the negative gate voltage is -4.5V. edit: It DOES NOT exceed max rating

[gkasprow]

here one can see interlock activation - violet trace

[gkasprow]

edit. the negative gate voltage is 10V, so still within the limit. The same as positive one

[hartytp]

hmmm...so no evidence of anything that could cause damage yet...

[hartytp]

Any other ideas for things to look at?

[gkasprow]

I'm curious why the FET output has such strange shape.

[hartytp]

Thanks Greg!

[gkasprow]

It's quite possible that I'm working with already partially damaged amplifiers. There is a chance that one can get broken after the first power-on cycle. I'll order a bunch of new ones and replace them. I run out of ADL ones while still have the other ones in my lab.

[hartytp]

Maybe. Something is happening that systematically damages these amps. I’ve never seen anything like it

[hartytp]

Before we get too convinced the amps are dying are we sure it’s not something like: mechanical issue creating a high Z intermittent short; problem with the bias; sequencing issue (eg input applied before bias); problemmwith matching circuit element; etc etc

[hartytp]

If we take a “damaged” component out and put it on an eval board/deadbug does it still not work?

[gkasprow]

I replaced the amp that died yesterday and left the booster working with full power over the night with scope connected and triggered by the output signal. If it dies we have a chance to catch the cause. The dead chip has 5V at its output and 0 at its input. I didn't throw it away so I can check it tomorrow. Today I had plans to do a lot of work but unexpected paperwork appeared. The report deadline was today :)

[hartytp]

Ok. Hopefully tomorrow is our day. Is there anyone else at wut/cti who could lend a fresh out of eyes?

[gkasprow]

Once I get new ICs, I will add 12V TVS and will see where the clamping occurs.

[gkasprow]

My colleague, Wojtek, was looking at it today and proposed a few ideas to study.

[gkasprow]

the ADL5536ARKZ are out of stock in Farnell :(

[hartytp]

Good. Remember zotino? How many times did we look at that data sheet and miss the ep connection to vss? SOmetimes a fresh pair of eyes is needed

[gkasprow]

Yes, I remember. I will pass by Technosystem tomorrow morning, maybe they have some on stock.

[hartytp]

Great, thanks!

[gkasprow]

The booster survived whole night running with full power. I have new ADL amplifiers so will replace it, add diode and check how it behaves with pulse input signal.

[hartytp]

Okay good! Also worth checking turn on/off standby (with/without signal applied) etc to see if there is any way to get a damaging transient

[gkasprow]

I will do it in the evening.

[hartytp]

:(

[gkasprow]

I have to leave for the meeting. Hope to be back at 5pm

[gkasprow]

full startup sequence with RF signal CH1 27 CH2 power stage gate CH3 first stage output CH4 RF channel output

[gkasprow]

The RF signal is on before the power stage supply

[gkasprow]

the off sequence

[gkasprow]

Booster power on

[gkasprow]

Brutal power off while working

[gkasprow]

Even during brutal power off, the correct off sequence is preserved.

[gkasprow]

The chip that was dead yesterday, connected to the 5V lab bench supply seems to be alive. It consumes 130mA of power and has 2.7V bias voltage. It looks like the soldering process took him back to life.

[gkasprow]

Another one that was changing the gain itself consumes 110mA of power and also has 2.7V input bias voltage

[hartytp]

Interesting! Maybe it’s some other problem like a short on the board or a failure of another component

[gkasprow]

I replaced the amp with a brand new one. Before powering it on, I added 12V Zener in series with LL4148. Activation of the interlock: The oscillations come from FM transmitter nearby.

[gkasprow]

This is interesting. With the clipping circuit, it achieves 11V. But this is with LL4147 capacitance and the probe capacitance.

[gkasprow]

more detailed view I added traces CH1 - ADL output CH2 - RF switch control signal CH3 - gate voltage CH4 - RF output

[hartytp]

@gkasprow thanks for posting that data.

In the first plot: https://github.com/sinara-hw/Booster/issues/329#issuecomment-562088019

• I assume the blue curve (PA gate voltage) has inversion on the scope? Otherwise the bias looks positive!
• So the RF reaches the power FET after the gate is biassed but before the drain voltage is applied. That can't cause problems can it?

[hartytp]

I don't understand the second plot https://github.com/sinara-hw/Booster/issues/329#issuecomment-562089490 (off sequence)

• why is there a shift in the gate DC level when the RF stops?
• why is the gate voltage decreased before the 29V current is disabled? Couldn't that cause a very rapid discharge of the capacitors through the FET leading to damage?

[hartytp]

i.e. can you have a look at the sense resistors and see if there are current spikes during turn on/off?

[hartytp]

I assume the blue curve (PA gate voltage) has inversion on the scope? Otherwise the bias looks positive!

Aah, no! I'm being silly. There is no inversion; when the unit is off the gate voltage is decreased to the max negative value (~-5V) to pinch off the drain current.

[gkasprow]

Exactly

[gkasprow]

Dead chips tha come back to life after heating are not a new thing. Some kind of damages in silicon can partially be reverted thanks to annealing process.

[hartytp]

I would have thought that gently removing an IC wouldn't get it hot enough to anneal the silicon?

[hartytp]

Anyway, even if that's so, it still doesn't explain how the IC was damaged.

I still wonder if the ICs are actually fine and it's something else on Booster that's the problem.

[gkasprow]

Well. I observed lack of gain and wrong dc signal. Supply was ok. DC drop on coil was 22mV.