Booster: firmware review

[hartytp]

What's the current status of the firmware that's loaded onto Booster at the moment? Should the instructions on the Wiki all work at the moment?

To check I understand things:

• ENABLE/DISABEL controls the biasing for the power FET. The pre-amp circuitry remains powered all the time
• START/STOP control the input switch (ANDed with the INTERLOCK status)
• The current and voltage measurement are measuring the bias voltage and current for the power FET (why are there two per channel?)
• What are the commands to measure the power (input, output forwards and output reverse) power for each channel?

Also, I need a way of finding the MAC address of booster (our department does MAC address whitelisting). How can I do that? Can I do that via the UART (e.g. by sending it a request, or if it prints out a boot message)?

[hartytp]

Can you remind me how the interlock works at the moment, please (maybe add a note to the wiki)? IIRC, we agreed that:

1. There is a hardware interlock that trips if the FET temperature gets too high
2. there is a hardware interlock that trips if the input, output forwards, or output reverse power is too high
3. there is an interlock that trips if the FET current/voltage is too high

Is that correct?

IIRC, we agreed that the set-point for (2) would be adjustable from firmware (e.g. to prevent damage to a sensitive load such as an optical modulator). Has this been implemented? If so, what is the command to change the set-point?

[hartytp]

Will all settings be saved to non-volatile memory? If not, can we have an option to save all settings, please?

My concern here is that if I have a sensitive load like an AOM, I may want to use the interlock to ensure that the amplifier can never drive too much power into the load and damage it. If the interlock settings are stored in volatile memory then it would be easy for a short power glitch to clear them without the user realising. This is prevented by having a means of saving them to non-volatile memory.

[hartytp]

One final request ;)

• if I understand things, the microprocessor continuously measures the drain current on each channel and feeds back on the gate voltage to keep the current at a set-point
• as discussed before, it would be good to have a way of changing the current set-point for each channel. The use-case for this is improving the power efficiency at lower powers by reducing the bias current (at the expense of lower gain and P1dB at lower bias currents).

I'm planning to make some measurements on P1dB and third-order intercept on Thursday or Friday, and would be interested to play around with this then while I have the equipment set up if that's possible?

[hartytp]

@gkasprow I'd like to test the FW/interlock later this week. Can you give me any feedback on the questions above?

Thanks!

[gkasprow]

@wizath please respond to @hartytp AFAIK at the moment the processor does not measure the current nor compensate the voltage because according to Maciej this is not necessary. There is only over-current protection implemented for every RF module supply rail.

[hartytp]

AFAIK at the moment the processor does not measure the current nor compensate the voltage because according to Maciej this is not necessary. There is only over-current protection implemented for every RF module supply rail.

Thanks for clarifying.

If possible, I'd definitely like a way of adjusting the bias current to play around with getting higher power efficiency at lower P1dB.

[gkasprow]

It is possible, but can also kill the output MOSFET :)

[hartytp]

You mean if the bias current is set too high? Can we make sure that the FW only allows the users to program it to values within a safe range?

[hartytp]

Anyway, I'm okay if this particular function is a "use at your own risk" one (MOSFETS are cheap anyway).

[gkasprow]

Yes, but the FW would have to monitor the supply current to be 100% sure

[hartytp]

@gkasprow Okay, well playing around with biasing is the lowest item on my to do list. If you can sort the other things out then I'll be happy.

In the long-term, adjusting the biasing is something I'm keen on to improve the power efficiency for lower output powers (0.5W-2W), so I'd like to find some way to do this in the FW. But, we can do that later on.

[hartytp]

ping @wizath I'm ready to play with the firmware/interlock whenever you can give me some feedback on the above questions. Thanks!

[wizath]

ENABLE/DISABEL controls the biasing for the power FET. The pre-amp circuitry remains powered all the time START/STOP control the input switch (ANDed with the INTERLOCK status)

In current version there is single button to power the channel LDO's, amp will start running after enabling power, because of (past) issue of protection circuit.

The current and voltage measurement are measuring the bias voltage and current for the power FET (why are there two per channel?) ADC on each channel gather information about: 30V current, negative voltage current and 5V current.

What are the commands to measure the power (input, output forwards and output reverse) power for each channel?

Measurement are displayed on VUART interface in 1s intervals. Optionally you can use SCIPI commands listed in Booster's wiki page.

Also, I need a way of finding the MAC address of booster (our department does MAC address whitelisting). How can I do that? Can I do that via the UART (e.g. by sending it a request, or if it prints out a boot message)?

You can type ifconfig to check MAC address

IIRC, we agreed that the set-point for (2) would be adjustable from firmware (e.g. to prevent damage to a sensitive load such as an optical modulator). Has this been implemented? If so, what is the command to change the set-point?

At this time, value is fixed within software, we could add single command to change DAC voltage for specified channel.

Will all settings be saved to non-volatile memory? If not, can we have an option to save all settings, please?

This hardware revision is not equipped with any kind of EEPROM memory, but AFAIK @gkasprow added the storage for next rev.

[hartytp]

@wizath Thanks for the feedback. Following up from your answer:

1. Is the list of commands on the Wiki up to date? Are all commands listed there currently implemented in the firmware, and are all commands in the firmware listed there?
2. Please can you add a *IDN? type command that prints a device identification string, complete with HW and FW revisions?
3. Is the input protection circuit implemented in the current firmware? If not, please can I have a version of the firmware with it implemented (the hardware issue is now fixed on my boards)?
4. Please can you clarify exactly what interlock is implemented in both HW and FW. (input power? reverse output power? output power? amplifier temperature? etc)
5. what does CHANnel:ENABle do? Does this turn on the LDOs + FET biasing?
6. what does CHANnel: START do? Does this clear the interlock?
7. what does CHANnel:MEASure:VOLTage? do? (The description on the wiki at the moment is confusing)
8. what does CHANnel:MEASure:CURRent? do? (The description on the wiki at the moment is confusing)
9. What exactly do the two buttons on the front panel do at the moment?
10. In the next firmware version (when will this be ready?), please could you add a command to set the FET bias voltage. Please add some basic checking to this command to ensure the the FET voltage can only be set within a sensible range that is unlikely to cause damage.

[hartytp]

11. How do I program/query the various interlock thresholds? e.g. how can I set what maximum output power the device can produce?

I want to test the interlock next, but need answers to (3) (4) and (11) before I can do that.

[wizath]

Is the list of commands on the Wiki up to date? Are all commands listed there currently implemented in the firmware, and are all commands in the firmware listed there?

Yes, they are up-to-date

Please can you add a *IDN? type command that prints a device identification string, complete with HW and FW revisions?

If I recall correctly it's implemented

Is the input protection circuit implemented in the current firmware? If not, please can I have a version of the firmware with it implemented (the hardware issue is now fixed on my boards)? Please can you clarify exactly what interlock is implemented in both HW and FW. (input power? reverse output power? output power? amplifier temperature? etc)

At this moment the ONOFF is shorted to 3.3V so any protection won't work. There is one hardware protection circuit (DAC) and I can implement a SW one, but need to know the voltage thresholds to disable the AMP.

what does CHANnel:ENABle do? Does this turn on the LDOs + FET biasing?

Enables power for selected channel

what does CHANnel: START do? Does this clear the interlock?

Enables ONOFF signal for selected chanel

what does CHANnel:MEASure:VOLTage? do? (The description on the wiki at the moment is confusing) Dumps the channel voltage measurements from forward and reverse power measurements

what does CHANnel:MEASure:CURRent? do? (The description on the wiki at the moment is confusing)

Dumps the channel ADC values for current measurements. These commands were implemented at early stage and weren't really consulted with anyone.

What exactly do the two buttons on the front panel do at the moment?

The first one just enables the power and since ONOFF signal is shorted to VCC so it's starting automagically. The fix is in comment here https://github.com/sinara-hw/sinara/issues/460#issuecomment-383871185 When I will get back working module I'll change the behaviour (1st will enable all channels and 2nd clear errors?)

In the next firmware version (when will this be ready?), please could you add a command to set the FET bias voltage. Please add some basic checking to this command to ensure the the FET voltage can only be set within a sensible range that is unlikely to cause damage.

I want to fix this fan ghosting and several other small issues, I think that will go around next week. Also, could you specify what commands would you like to get from this SCIPI interface and I'll prepare them for you.

[hartytp]

@wizath Thank you for the response.

Before I reply to your questions, I'd like to confirm with @gkasprow how the interlock is supposed to work. @gkasprow is this correct:

• We have a hardware interlock for the input power and the output forward power. So, no software interlock is needed here.
• We do not have a hardware or software interlock on the reverse power. An interlock is not needed on the reverse power because the amplifier can handle full reverse power (all phase angles). Is this correct? Or, do we need to implement a reverse power interlock? If we do need an interlock on the reverse power, will it be done in software, or will we add a DAC/fixed potential divider to do it in hardware?
• There will be an interlock on the temperature of each channel implemented in firmware
• There is an interlock on the bias current for each channel implemented in software? If we use feedback to stabilise the bias current then this interlock should shut things down if the feedback loop rails (i.e. if the programmed bias voltage goes outside of the safe range).

Is that all correct? Are there any other interlocks that I've missed? Or, is that the complete list?

I know we've talked about this a few times before, but I wasn't sure if we reached a conclusion. Let's finalise this once and for all and I will write it up on the Wiki.

[hartytp]

@gkasprow I know this is discussed in https://github.com/sinara-hw/sinara/issues/460#issuecomment-387895107, but I would really appreciate it if you could have a look at my post above and give me feedback. Is what I said there correct? Is there anything I'm missing?

[hartytp]

@gkasprow What I want to get clear on first is what our long term plans are here (i.e. not necessarily what has been implemented in the current firmware/hardware).

Also, what is our plan for the bias current/voltage? Will we use feedback to control the current? Or, just run this open loop?

[hartytp]

ping @gkasprow

[hartytp]

ping @gkasprow Please can you confirm what the interlock/biasing plans are? Once you do that, we can finalize a specification for what the firmware must do, and @wizath can start work on the next version.

I'm keen to get this done asap, because I can't test the interlock until it's done.

[hartytp]

@gkasprow ping!

Please can you confirm how the interlock and biasing are supposed to work so that we can finalise the plans for booster. I'd really like to agree on the specification for the firmware before I leave on Wednesday morning.

[gkasprow]

There are following interlocks:

• each current that supplies each module module is monitored by fast ADC that has digital thresholds and can generate alert when current is exceeded. This happens without software intervention. Each output is routed to the CPU. At the moment we did not implement firmware to react on such event.
• each module has a DAC that sets a thresholds for input power and reflected output power. Each time the power is exceeded, the input gets disconnected and CPU is notified about it. User can resume operation by pressing front panel button. CPU knows what caused the alert. Your modules has this interlock blocked due to transient, you can enable it by removing red wire and adding R + C.
• in addition, input power is measured by slow ADC. According to Tomasz, this would be needed during calibration. It is not supported by SW.
• There are temperature sensors placed on the aluminium rail that holds the amplifiers. I could not find better place for it. Ideally they should be screwed to the amplifiers but this makes problem with amplifier replacement. I designed long PCB that makes assembly of the sensors easier without soldering individual wires. 6 temperature sensors control speed of 6 fans. There is an averaging mode in the temperature regulators that should eliminate haunted noise of the fans. I wanted to make sure that fans will switch on without software intervention and so the thermostat modules do.
• There are temperature sensors placed inside of the PA module. They are read by the firmware.
• We planned to implement real time SWR protection that continuously monitors this parameter. But this is problematic with calibration and finally we won't implement it. Anyway, user can read the ADC results and do proper calculation and calibration
• There is possibility of implementing bias current stabilisation, but this needs some kind of PID regulator. The CPU has all information needed. User should not be surprising by changing bias current in random way during experiment. One can discover new physics in this way :)

[gkasprow]

So maybe such bias stabilisation should be made on request ?

[hartytp]

Thanks @gkasprow!

each module has a DAC that sets a thresholds for input power and reflected output power.

Is that correct? Looking at the schematics, it seems the it is the input and output forward power that are interlocked in hw. The ret detector Q/Q_n outputs are nc atm.

[hartytp]

each current that supplies each module module is monitored by fast ADC that has digital thresholds and can generate alert when current is exceeded. This happens without software intervention. Each output is routed to the CPU. At the moment we did not implement firmware to react on such event.

This seems worth implementing, doesn't it? Shouldn't the behaviour be to shut the RF switch and disable the regulators for the channel in case of an over current situation?

[hartytp]

There are temperature sensors placed inside of the PA module. They are read by the firmware.

Should we place a software protection here that disables any channel whose temperature exceeds the safe maximum?

[hartytp]

There is possibility of implementing bias current stabilisation, but this needs some kind of PID regulator. The CPU has all information needed. User should not be surprising by changing bias current in random way during experiment. One can discover new physics in this way :)

:) indeed

This is my fear with this kind of feedback loop. Passive stability is obviously better. The question is just how good the passive stability is, and whether active stabilisation of the bias current is necessary. Looking at the bias voltage/current plot for the FET we're using, it seems that it is quite sensitive. So, my concern is that we may see large power changes at the amplifier gets hot.

If the DAC that controls the bias current has a decent resolution and proper filtering on its outputs, I wouldn't expect it to add significant noise to the RF output.

[hartytp]

each module has a DAC that sets a thresholds for input power and reflected output power.

By the way, if you don't think we need reverse power interlock then it's okay with me to leave it out. So long as the amplifier can handle full reverse power, I think the current situation is okay. I just want to make sure that I understand the design decisions.

[gkasprow]

Look here https://github.com/sinara-hw/Booster/releases/download/v1.1_rc1/PA_8CH_CTRL.PDF The IC5 is the DAC that sets the threshold value for the detectors that are sensing input and reflected power on the bottom board.

[hartytp]

@gkasprow I think there are two questions here:

1. What is implemented in the current hw?
   • IC5 supplies two thresholds IN_VREF and RF_VREF
   • The PA schematic (https://github.com/sinara-hw/Booster/releases/download/v1.1_rc1/PA_Channel_SUP_CTRL.PDF) shows that FW_VREF is connected to the forward power detector, with the threshold for the reverse directional coupler not connected. Assuming this schematic is correct, it looks like there is currently no (hardware) reverse output power interlock.

[hartytp]

2. What should be implemented:
   • We have two goals here: protect the amplifier and protect the load
   • To protect the load, we only need to limit the output forwards power. In principle, this can be done by interlocking the input power. In practice, it seems much better to limit the output forwards power. This way, variations in the circuit gain (for example over frequency, between channels or due to amplifier saturation) don't need to be calibrated out to provide an accurate limit on the output power, which is needed to protect delicate instruments.
   • To protect the amplifier itself, we need to limit the input power. Again, this could be done by limiting the output power only, but it seems to be safer to limit the input as well (although, if you're happy not having a hardware interlock on the input then that's fine by me).
   • To protect the amplifier itself, we may also need to limit the output reverse power. This just depends on whether the amplifier can handle full reverse power without damage/instability. If the amplifier can handle full reverse power then we do not need a reverse power interlock. I leave this decision up to you (but please let me know your decision).

[hartytp]

So, assuming that the amplifier can handle full reverse power, it seems that the current PA schematic -- which has input and output forwards power interlocks -- is the correct way to do it. But, please let me know what you think of my comments above.

[gkasprow]

I'm a bit surprised that the module has only the FWD power monitoring realised in HW. I will clarify it with Maciej. I've just looked at the Tomasz design and it is done in the same way. The original idea was to have this protection realised in the firmware so we will probably have to do it now. Since it is constant threshold protection and we care mainly about thermal effects of handling full reflected power by the output amplifier, implementation in firmware will be easy. Delay of a few tens of ms won't kill the output stage.

[hartytp]

I'm a bit surprised that the module has only the FWD power monitoring realised in HW. I will clarify it with Maciej.

As I said above, I think that hardware interlocks on the input and output forwards power are more important than the output reverse power, so I'm happy with the current situation so long as you and Maciej are -- although, please make sure that the labelling on the various PCBs is consistent.

Since it is constant threshold protection and we care mainly about thermal effects of handling full reflected power by the output amplifier, implementation in firmware will be easy

Is it thermal protection or over voltage protection? I know that in some cases, reverse power can kill amps very quickly by creating large voltages at the FET output. In this case, the protection needs to trigger quickly. Anyway, not sure what the damage mechanism is in this amplifier, and I'm happy to do what you and Maciej think is best.

The original idea was to have this protection realised in the firmware so we will probably have to do it now.

Sounds good. However, don't bother implementing this unless you think it's necessary. e.g. if the damage mechanism is thermal, will the thermal overload protection you've implemented be enough? Also, if the amplifier can take full reverse power anyway, we don't need an interlock on this, since it's safe by design!

[hartytp]

Given the above discussion, here is my summary/proposal for the interlock and firmware interface. Anyone involved/interested in this project (particularly Greg/Maciej/Tomek) should feel free to comment.

Interlocks

• We will have a software interlock on the power FET bias current for each amplifier. This ensures that variations in the temperature/bias voltage cannot lead to excessive bias current flowing. The threshold for this will be chosen by the circuit designers and hardcoded into the firmware.
• We have fast (hardware) interlocks on the input and output forwards power. These protect both the amplifier and the load. The threshold is settable in software, although the threshold cannot be increased above a maximum safe level, chosen by the circuit designer and hard-coded into the firmware.
• The output reverse power is monitored by ADCs. There will be a software interlock used to limit it.
• The temperature of each amplifier channel is monitored by software. If it exceeds a (hard-coded) maximum value then the channel is shut down (bias current cut off and switch closed)

If any of these interlocks trip, the input switch is shut off (input disconnected) and the power FET bias current/voltage is shut off.

Firmware commands to follow...

[hartytp]

Firmware commands

• All commands either return OK or an error string.
• *IDN? returns an ID string, including the hardware and software revision numbers
• INTerlock:INPut <channel> <power> max input power in dBm, e.g. INT:INP 0 -5 to set the input power interlock threshold for channel 0 to -5dBm. If the threshold is above a hard-coded (please document what the threshold is) maximum, an error is returned and the threshold is not adjusted. INTerlock:INPut? <channel> returns the input power interlock threshold for a given channel in dBm.
• INTerlock:OUTput <channel> <power>' max output power in dBm e.g. 'INT:OUT 0 33 to set the output power interlock threshold for channel 0 to 33dBm. If the threshold is above a hard-coded (please document what this max is) maximum, an error is returned and the threshold is not adjusted.
• INTerlock:OUTput? <channel> returns the output forward power interlock threshold
• ~`INTerlock:SAVe' save all interlock thresholds to non-volatile memory.~
• INTerlock:REVerse <channel> <power>' max output reverse power in dBm e.g. 'INT:REV 0 33 to set the output reverse power interlock threshold for channel 0 to 33dBm. If the threshold is above a hard-coded (please document what this max is) maximum, an error is returned and the threshold is not adjusted.
• INTerlock:REVerse? <channel> returns the output reverse power interlock threshold
• INTerlock:STAtus? [ALL]|<channel>' returns the interlock status (1or0`) for either 1 channel or for all channels (and-ed together).
• INTerlock:CLEar [ALL]|<channel> resets the interlock for either a given channel or for all channels
• MEAsure:TEMperature? <channel> returns the temperature for a given channel in C
• MEAsure:CURrent? <channel> returns the bias current for a given channel in amps
• MEAsure:VOLtage? <channel> returns the bias voltage for a given channel in volts
• MEAsure:INPut? <channel> returns the input power for a given channel
• MEAsure:OUTput? <channel> returns the forward output power for a given channel
• MEAsure:REVerse? <channel> returns the output reverse power for a given channel
• CHAnnel:ENAble [ALL]|<channel> <enabled> enables/disables (1 or 0) the power FET bias current/voltage for a given channel or for all channels
• CHAnnel:ENAble? [ALL]|<channel> returns 1 if the bias for a given channel is enabled or 0 if it is disabled. If ALL (default) then it returns the status for all channels anded together
• CHAnnel:BIAS? <channel> returns the bias voltage for a given channel
• CHAnnel:BIAS <channel> sets the bias voltage for a given channel. If this is outside the safe range (hard-coded into the firmware) then an error is returned and the bias setting is not changed. Note that there is additional protection when changing the bias voltage since the firmware interlock continuously monitors the bias current.

@wizath @gkasprow Please feel free to make suggestions for how this should be changed. Also, please feel free to change my notation if I haven't used standard SCPI notation, or to suggest better ways of doing any of this!

Please make sure to document things like the safe range for the bias voltage etc.

Edit/later additions to the above:

• 'RESet' reset all settings to factory defaults
• On second thoughts, let's make it so that changes to bias and interlock thresholds always save to non-volatile memory and remove the save command.

[hartytp]

Front panel buttons

• One button clears the interlock for all channels. Equivalent to INTERLOCK:CLEAR
• One button toggles between enabling all channels enabled and all channels disabled (equivalent to CHANNEL:ENABLE 1 or CHANNEL:ENABLE 0)

[hartytp]

@gkasprow @wizath and others, please offer feedback on this. Once we finalise this all, I'll post it to the Wiki. Thanks!

[gkasprow]

I talked with Tomasz and he says that the hardware interlock was on the reflected power! He simply swapped outputs on his coupler and did not modify schematics because he does not know how to use Altium. And Maciej simply duplicated the error. So we have to switch the coupler outputs in next revision. We can also swap them in current revision.

[hartytp]

He simply swapped outputs on his coupler and did not modify schematics because he does not know how to use Altium. And Maciej simply duplicated the error.

:(

So we have to switch the coupler outputs in next revision. We can also swap them in current revision.

You mean switching them so that the hardware interlock applies to the forward power, and not to the reverse power?

Yes, please can you do that: switch the coupler outputs in both the current prototypes (so that we can test them) and in the next revision so that the input power and output forwards power are hardware interlocked.

[hartytp]

Other than that, do you agree with everything I posted above?

[dnadlinger]

(@hartytp: This explains my confusion, resp. the reason I asked you about the hardware interlocks a couple of weeks back. Agreed that input/output forward power hardware interlocks are the way to go, together with whatever we need to do on the reverse power to tolerate infinite opens/shorts on the outputs.)

[gkasprow]

A the moment we have HW interlocks on the input power and the output power. Original intention was to make it on the reflected power. So we leave it as it is?

[hartytp]

In that case, yes, please leave it as it is.

Please can you double check that you are happy the descriptions I posted above for the interlock/firmware/button functions. And double check that things are implemented that way in the HW/FW.

Also, before we finalise the next revision, please make sure that all schematics are updated and consistent.

Thanks!

[hartytp]

@gkasprow are there any other things we should be able to query from the firmware for diagnostic purposes? All ADCs that we have in the hardware should be readable via the ethernet interface for debugging.

[gkasprow]

@hartytp this looks like a complete list. At the moment we don't have non volatile memory on the main board - only small ID chips on the modules.

[hartytp]

@gkasprow thanks for confirming. I'll copy that to the Wiki at some point in the next few days when I have time.

When will @wizath have time to implement that? I'd ideally like someone to test out Booster with the complete firmware before we order the next hardware revision.

At the moment we don't have non volatile memory on the main board - only small ID chips on the modules.

I think that non-volatile storage of the interlock settings is essential. Otherwise, they can't be relied on to protect sensitive loads. So, we have two options:

1. If the EEPROM you will use in the next version comes in a sensible package (eg SOIC) then can we kludge one onto the current hardware and dead bug it?
2. If we can't dead bug it then we can complete our testing without the memory chip and order the next hardware round. Once the next hardware round has been completed, we'll upgrade the control board in the unit we already have. Before this upgrade, we can't rely on the adjustable interlock thresholds.

Let me know which one you want to do.

[gkasprow]

Dead bug is OK, we have free CPU pins.

[hartytp]

Cool, let's do that.