Channels show error condition on power-up without log messages

[cjbe]

After power-cycling the Booster, channels 0 and 4 reliably (N=5) go into "error" after ~2s (the other channels show "interlock tripped"). There are no errors shown via logstash.

There are clearly some bad calibrations being used, seeing as all of the DAC2 (forward power interlock) are set to 0 (due to #182). Are there some other problems caused by this as well? I don't understand how these channels can go into error without something being present in the log

> logstash
[INFO] hardware revision 3
[INFO] device boot
[INFO] SYSCLK frequency: 168000000
[INFO] PCLK1 frequency: 42000000
[INFO] PCLK2 frequency: 84000000
[INFO] ADC: OK | raw: 1981 | VrefInt 1.21 V
[INFO] PGOOD: OK
[INFO] Found mainboard EEPROM, loading values
>

PGOOD: 1
FAN SPEED: 30 %
AVG TEMP: 0.00 CURRENT: 0.00
CHANNELS INFO
==============================================================================
                #0      #1      #2      #3      #4      #5      #6      #7
DETECTED        1       1       1       1       1       1       1       1
HWID            B2:13   B1:CF   90:A2   69:E0   6D:76   56:81   3C:9C   39:6B
TXPWR [V]       0.01    0.03    0.01    0.02    0.01    0.01    0.01    0.01
RFLPWR [V]      0.01    0.01    0.01    0.01    0.01    0.01    0.03    0.02
TXPWR [dB]      -7.29   -7.33   -6.30   -6.99   -7.61   -7.45   -7.78   -6.78
RFLPWR [dB]     -2.79   -2.10   -3.55   -4.31   -4.44   -5.12   -5.83   -5.78
I30V [A]        0.094   0.015   0.011   0.013   0.002   0.013   0.011   0.003
I6V0 [A]        0.150   0.252   0.277   0.500   0.088   0.257   0.250   0.267
5V0MP [V]       4.986   4.977   4.968   4.976   4.960   4.965   4.953   4.976
ON              0       1       1       1       0       1       1       1
SON             0       0       0       0       0       0       0       0
IINT            0       0       0       0       0       0       0       0
OINT            1       1       1       1       1       1       1       1
SINT            0       0       0       0       0       0       0       0
OVC             1       0       0       0       1       0       0       0
ADC1            10      56      11      35      11      23      12      13
ADC2            10      10      10      11      11      13      41      33
DAC1            4095    4095    4095    4095    4095    4095    4095    4095
DAC2            0       0       0       0       0       0       0       0
SCALE1          83      87      87      85      85      86      86      87
OFFSET1         615     695     558     605     658     661     681     599
BIASCAL         1811    1960    1690    1700    1712    1687    1657    1756
HWIS            0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
HWIO            0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
LTEMP           25.00   25.25   25.00   25.00   26.00   25.25   24.00   25.00
RTEMP           25.25   25.50   26.00   25.25   24.00   25.50   25.25   25.50
==============================================================================

[wizath]

Is it happening while channels are powered on?

[cjbe]

Yes. I can reproduce this by power cycling the Booster, then pressing the "standby" button to enable all the channels. All the channels go into "interlock tripped" (due to #182), then after ~2 seconds channels 0 and 4 go into "error"

[wizath]

Could you try setting the OVC limits higher - command ovc <channel> <value>. Default value is 51 for revision v1.3.

[cjbe]

OK - what are the units, and how much of a increase is sensible?

[wizath]

It's like 10~ == 100 mA

[cjbe]

This does not seem to solve the problem. I have set the OVC to 60 and I still see the channel tripping into "error" frequently when the interlock trips.

After fixing #182 the symptoms of this have changed somewhat. The channel stays OK after power-up, but trips into "error" most times the interlock trips. Before the channel goes into error the SCPI command INT:STAT? returns something funny (16, rather than 0 or 1). Shortly after this the channel trips into error - the time ordering of this can be seen via logstash:

[INFO] network client 10.255.6.32 connected
[ERROR] Interlock tripped on channel 4, i=0 o=1
[INFO] network client disconnected
[ERROR] OVC on channel 4, current 0.00

My code disconnects when I get the odd response from INT:STAT?, then a fraction of a second later the channel trips into error.

[wizath]

Disabled OVCurrent Protection. Could you try if it helps? Sorry but I was overthinking with that ovc command, since on startup channel overwrite changes made by that command.

eth-scpi.dfu.zip

[cjbe]

@wizath okay, nearly there...

That fixes the OVC issue. However, on channel 4 when the interlock trips 'INT:STAT?' returns 16. That makes sense as 1 <<4. But, in all other cases it was returning 1 or 0 for the channel. So, I think there is an inconsistency in the code. Can you have a look at that?

[cjbe]

This is important for us, as we're using some automated tooling to test Booster and these kinds of inconsistencies break our code.

e.g.

Testing channel 4 ...
Gain:
40.5 dB (at P_out = 20.0 dBm)
39.3 dB (at P_out = 30.0 dBm)
38.6 dB (at P_out = 35.0 dBm)
Internal power meter:
f=100 MHz: error -0.2 dB at P_out = 20.0 dBm
f=100 MHz: error -0.1 dB at P_out = 30.0 dBm
f=100 MHz: error -0.0 dB at P_out = 35.0 dBm
f=250 MHz: error 0.0 dB at P_out = 20.0 dBm
f=250 MHz: error 0.2 dB at P_out = 30.0 dBm
Traceback (most recent call last):
  File "test_channel.py", line 29, in <module>
    p_in, p_out = set_output_power(p, ch=args.channel, freq=freq, iface=iface)
  File "/home/ion/scratch/cjb/booster_tests/set_output_power.py", line 29, in set_output_power
    tripped = not iface.booster.get_interlock_status(ch)
  File "/home/ion/scratch/oxart/oxart/devices/booster/driver.py", line 104, in get_interlock_status
    return not self._query_bool("INT:STAT?", channel)
  File "/home/ion/scratch/oxart/oxart/devices/booster/driver.py", line 72, in _query_bool
    "Unrecognised response to {}: '{}'".format(cmd, resp))
Exception: Unrecognised response to INT:STAT?: '16'

[wizath]

Sorry, that was one command that ran away from my last refractor :disappointed:. Fixed it to work like other commands e.g: int:stat? all returns bitmask with bit shifter statuses for each interlock int:stat? 1 returns 0 or 1

https://github.com/sinara-hw/Booster/issues/98#issuecomment-475288436