Specification

[kaolpr]

Introduction

In complex control systems, aspect of firmware updates distribution is critical for system development and maintenance. Due to the limited accessibility of some installations or their spacial extent ability of remote update is a vital feature.

This project concentrates on Advanced Mezzanine Cards used in mTCA environment, though it can be used for any FPGA-based card with network connectivity and supported FPGA.

Some FPGA AMC modules that do not have Ethernet-enabled MMC, but do have a connection between FPGA and mTCA backplane port 0/1:

• AFC,
• AFCK,
• Sayma AMC

Those modules require a special FPGA user-application to be able to perform a network update. Such an application would basically be the same for every card. The only differences would result from FPGA family and board architecture, however principle of operation would be the same in almost any case. This project aims to provide such an application and make it independent from the specific application developed by the end-user.

Concept of operation

FPGA boots from the Golden Image that is provided by this project. It connects to the external server and checks for the update. If it's not available, FPGA is rebooted into the end-user application. However, if an update is available on the remote server it is downloaded, verified, and written to the configuration memory.

This application note can be used as a reference.

As handling network traffic is rather cumbersome in HDL, it is suggested that a soft-core approach should be taken. Litex provides a complete framework for building soft-core based systems. It provides both IP cores for various peripherals, notably Ethernet, DRAM of SPI flash.

LitexBuildEnv project provides an environment and instructions for using Litex and building software stack. Which stack to use is left to the designer. However, one should bear in mind that the output image should not be of an excessive size as it must fit in the flash memory along the end-user application.

Specification

1. The project should generate a bitstream that enables remote updates of the end-user-application stored in the card flash memory.
2. Ethernet should be used as an update medium.
3. The application architecture should make it easy to port to different cards (decouple board-specific code).
4. Initial target FPGA families: Xilinx Series 7, Xilinx Ultrascale+.
5. A specification (if commonly used technology like FTP or HTTP is used) or implementation (if commonly used technology can not be used) of firmware server must be provided.
6. A concept of end-user-application update channels must be implemented. It should allow end-user to define different channels serving different applications. Every card must be configurable to track a specific update channel.
7. The update procedure must be designed in such a way that it can not damage the golden image.
8. Update procedure must be designed in such a way that it ensures correct writing (no errors, checksum ok) of the updated application bitstream to flash memory.
9. The card should be able to be configured using a serial console to use either DHCP or static IP address.
10. The card should be able to be configured using a serial console to track the selected update channel.
11. Persistent storage should be implemented for storing configuration data in the flash memory.

[gkasprow]

Nearly the same approach is used in case of Marble It's worth checking the status of LBNL development.

[gkasprow]

@kaolpr can you make the repo more open, please ?

[ldoolitt]

Yes, the Marble board series intends to put into practice some concepts for remote updates that have been developed at LBNL over many years.

As handling network traffic is rather cumbersome in HDL

Besides LiteX/LiteETH, see Packet Badger. We've demonstrated that a build with about 1200 LUT is capable of getting on the network and programming its own flash.

Golden images are a great idea, and not new. See Remote FPGA Upgrades with Fail-Safe Booting, Qiang Du, Jack Olivieri, Lawrence R Doolittle, LLRF13, Lake Tahoe poster We at LBNL consider it essential to have the watchdog-based reboot-to-golden-image feature.

Every FPGA I know of, including Xilinx 7-series, can write its own boot flash. To write-protect a subset of that flash to hold a golden image, you need two things:

1. An understanding of the protection modes of your flash chip, and
2. Disable QSPI, so the WP pin of the SPI flash is under hardware control (hardware switch, not the FPGA).

LBNL has strong interests in collaboration and tech transfer. We're happy to help and communicate our experiences.

[gkasprow]

@ldoolitt that's great. We have a funding for this task and will assign an engineer for a few months to make it working reliably on all platforms.

[lerwys]

Hello @ldoolitt, @gkasprow, everybody,

We, at LNLS, are using AFC in production for some years now, but we only have PCIe (Ports 4, 5, 6 and 7) to the MCH, no AMC Port 0,1 connection to the MCH, because we made a choice to use these ports as P2P (custom backplane).

I don't know how feasible this is and it's probably be a big architectural change, but it would be great if this approach could also work with PCIe, somehow. I know, at least, two other institutes that use AMC boards with PCIe that could benefit from this, as well.

We could help with that if necessary, of course.

[ldoolitt]

A specification (if commonly used technology like FTP or HTTP is used) or implementation (if commonly used technology can not be used) of firmware server must be provided.

At LBNL we've used both custom application-specific protocol and TFTP. With no soft-core on the FPGA (pure fabric), we used a custom python tool to write the flash over the network. With a small soft-core on the FPGA, it could implement TFTP, and host TFTP software is easy to come by.

I recommend staying away from TCP in general; UDP is a much better match to an FPGA's strengths. But not raw Ethernet! That's a bear to work with on the workstation side, due to permissions and security.

[gkasprow]

one can have a look at the IP-bus protocol. Its widely used in HEP community

[kaolpr]

I recommend staying away from TCP in general; UDP is a much better match to an FPGA's strengths.

@ldoolitt You mean when dealing in fabric? I can't see much of a problem when we'll do it in soft-core, especially that our resources are not very limited.

I'd like to make the first approach using the LiteX ecosystem. It will give us a prototype of the golden image quickly. It may also lead a way to implement @lerwys suggestion of using different update channels, e.g. PCIe.

What, in your opinion (@lerwys, @ldoolitt), is the upper limit of the golden image bitstream size?

[lerwys]

I don't have a very strong opinion about this. For the AFC we have a 256Mb flash memory, so we could store 3 Artix7 200T bitstreams (77,845,216 bits according to 7 Series FPGAsConfiguration User Guide). With bitstream compression we can reduce the image size even more, I guess.

We don't foresee any usage for user data for this flash memory. So, for the AFC I don't think we need any constraints for the golden image size.

For boards using 128Mb flash memory I think there will be more compromisses, specially if they use Artix7 200T, for instance. In this case, allowing 2 complete application bitstreams + 1 golden image could be impossible. And then, we might need to use just a single application image + golden image. And, in this case, limiting the golden image to ~50Mb (128-78) could be necessary.

[ldoolitt]

I recommend staying away from TCP in general; UDP is a much better match to an FPGA's strengths.

@ldoolitt You mean when dealing in fabric? I can't see much of a problem when we'll do it in soft-core, especially that our resources are not very limited.

In fabric, definitely. Also a soft core that uses on-chip memory. If using TCP pushes you to demand off-chip memory, take a deep breath and think about my recommendation.

[ldoolitt]

one can have a look at the IP-bus protocol. Its widely used in HEP community

I looked at IP-bus once. It's cute, but defintely "modern" and designed by software people. That makes it llook less than attractive to a hardware person. It's not compatible with the in-fabric processing paradigm of my Packet Badger subsystem, because the responses are sometimes longer than the requests.