Invalid glasgow_config after flashing a modified firmware

[DurandA]

Hi,

I got an assembled Glasgow rev C1 and I am playing with the FX2 firmware. I noticed some strange issues if I erase the FX2 EEPROM, perform a glasgow factory --rev C1:

[ 2080.974006] usb 1-2: New USB device found, idVendor=20b7, idProduct=9db1, bcdDevice= 1.31
[ 2080.974011] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2080.974014] usb 1-2: Product: Glasgow Debug Tool
[ 2080.974017] usb 1-2: Manufacturer: whitequark research
[ 2080.974020] usb 1-2: SerialNumber: C1-20210212T011722Z

And then after a power cycle glasgow flash --firmware firmware/glasgow.ihex with a modified firmware:

[ 2281.649128] usb 1-2: New USB device found, idVendor=04b4, idProduct=8613, bcdDevice= 1.f9
[ 2281.649133] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2281.649136] usb 1-2: Product: Glasgow Debug Tool
[ 2281.649139] usb 1-2: Manufacturer: whitequark research
[ 2281.649141] usb 1-2: SerialNumber: O9-9999999999999999
[ 2286.717811] usb 1-2: can't set config #1, error -110

The board becomes unusable after that and requires a flash erase with an external tool as it is not properly enumerated after a power cycle otherwise:

[ 2624.222003] usb 1-2: new high-speed USB device number 28 using xhci_hcd
[ 2629.374022] usb 1-2: device descriptor read/64, error -110

However, the issue does not appear if I first flash the firmware compiled from master and then the same custom firmware.

My first thought that maybe a memory region grew a bit too much and the memory location of glasgow_config slightly moved and/or was erased by other regions. However it seems to be allocated the very end of the usable memory: https://github.com/GlasgowEmbedded/glasgow/blob/25fe8d326ef2cb01d1316136ec7a86967e90f21b/firmware/glasgow.h#L81

So I tried to slightly increase XRAM and reduce CODE by the same amount: https://github.com/GlasgowEmbedded/glasgow/blob/25fe8d326ef2cb01d1316136ec7a86967e90f21b/firmware/Makefile#L4-L6

None of my attempts solved the issue. Do you have any idea of what kind of black magic could be happening?

[attie]

Unfortunately I don't think custom firmware is going to receive much direct support from the Glasgow project. Sorry.

Depending on what you're doing, I'd encourage you to inspect the official firmware, how it is built, and work from there.

If you've taken the existing firmware and tweaked it, then the bcdDevice, Serial Number, and other fields are stored at a specific location in flash (as you've identified), and looking at your values, these all appear to be bad / wrong... Possibly due to erased flash with no data (does a bcd field mask / limit to 9?, wrt revision 'A' + 0xf - 1 == 'O'), possibly due to other data being placed there incorrectly.

This location is at the very end of the flash (0x4000 - CONF_SIZE = 16 KiB - 64 Bytes), and that 0x4000 / 16 KiB size directly relates the size of the FX2's internal RAM - i.e: your CODE + RAM + CONFIG cannot exceed this size.

If you have access, it might be worth dumping the flash and looking at what is stored there.

The USB enumeration / configuration change failing implies that something is wrong with your USB handling, but I can't offer much more than that unfortunately.

Have you confirmed that you can simply build the official firmware, and that it works correctly?

The board becomes unusable after that and requires a flash erase with an external tool as it is not properly enumerated after a power cycle otherwise:

It should be possible to short the two "REC" pins at power up, to present the FX2 as if it had no firmware - assuming you're on the revC1 board from me, the pins are here:

---

Edit: I just followed the link to your custom firmware and looked at what you're doing.

1. It looks like your changes are implementing support for the ATECC chip - if you're using the board from me without modifications, then please be aware that this chip isn't present.
2. The ATECC chip was removed from future revisions for various reasons, so I suspect that support for it isn't something we're looking for (in case you're aiming at a PR).

[electroniceel]

The board becomes unusable after that and requires a flash erase with an external tool as it is not properly enumerated after a power cycle otherwise:

It should be possible to short the two "REC" pins at power up, to present the FX2 as if it had no firmware - assuming you're on the revC1 board from me, the pins are here:

If flashing with the REC pads shorted doesn't work for you, it could be issue #252 you are experiencing. This happens when you have flashed firmware (and not just the VID/PID for the FX2). You need to flash basic firmware from libfx2 as outlined in that issue to work around that until a proper fix for #252 is implemented.

[DurandA]

Thank you very much @attie and @electroniceel for the help. I will have a closer look at it in the next few days.

1. It looks like your changes are implementing support for the ATECC chip - if you're using the board from me without modifications, then please be aware that this chip isn't present.

I'm using one of your boards :wink: and yes U34 is unpopulated. I soldered the SOIC variant on the test points at the back.

2. The ATECC chip was removed from future revisions for various reasons, so I _suspect_ that support for it isn't something we're looking for (in case you're aiming at a PR).

I am aware of it. I joined the #glasgow IRC last year to propose my help for authentication and @whitequark told me that the idea was discarded. I proceeded anyway since I am doing some academic work on IoT security and was interested in it.

Thanks for the REC tip, I did not notice it on the board nor in the schematic.

[electroniceel]

2. The ATECC chip was removed from future revisions for various reasons, so I suspect that support for it isn't something we're looking for (in case you're aiming at a PR).

I am aware of it. I joined the #glasgow IRC last year to propose my help for authentication and @whitequark told me that the idea was discarded. I proceeded anyway since I am doing some academic work on IoT security and was interested in it.

I'm curious to know why you decided to connect your ATECC to the internal I2C bus of Glasgow then and not to one of the regular ports A or B.

Using the regular ports would allow you to easily create a Glasgow applet for the ATECC or use a REPL script like https://github.com/GlasgowEmbedded/glasgow/blob/master/examples/i2c-pcf8574.py There is no interface for accessing the internal I2C yet, so all I2C transactions must be added to the firmware, as you probably already figured out. This makes it much more complicated, at least from my point of view.

[DurandA]

I'm curious to know why you decided to connect your ATECC to the internal I2C bus of Glasgow then and not to one of the regular ports A or B.

The main reason was that the board was already designed with the ATECC connected to the internal bus. Also I was able to start working on the firmware with a cheap FX2LP dev board as I did not have a Glasgow and was not very confident about assembling BGA parts myself. I also think it makes more sense to be able to authenticate the device at any stage without replacing the gateware.

Besides the issue when flashing this firmware, the minimal driver for the ATECC I created for the FX2 seems to work fine and have sufficient features for public key authentication and certificate retrieval. The Python authentication module is still WIP.

[DurandA]

I am back to firmware hacking and I am still annoyed by this issue. I am now stuck in a loop that prevents any useful work.

From a C1 Glasgow with a modified firmware:

[  980.665167] usb 1-2: new high-speed USB device number 24 using xhci_hcd
[  980.817066] usb 1-2: New USB device found, idVendor=20b7, idProduct=9db1, bcdDevice= 1.31
[  980.817071] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  980.817074] usb 1-2: Product: Glasgow Debug Tool
[  980.817077] usb 1-2: Manufacturer: whitequark research
[  980.817079] usb 1-2: SerialNumber: C1-20210330T015615Z

Then I send a USB requests that fails with usb1.USBErrorPipe: LIBUSB_ERROR_PIPE [-9]:

data = await device.control_read(usb1.REQUEST_TYPE_VENDOR, 0x20, 10, 0, 16)

After that the device is unusableso I disconnect and then reconnect the Glasgow board:

[ 1071.765476] usb 1-2: USB disconnect, device number 24
[ 1074.350063] usb 1-2: new high-speed USB device number 25 using xhci_hcd
[ 1074.502355] usb 1-2: New USB device found, idVendor=04b4, idProduct=8613, bcdDevice= 1.f9
[ 1074.502360] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1074.502364] usb 1-2: Product: Glasgow Debug Tool
[ 1074.502367] usb 1-2: Manufacturer: whitequark research
[ 1074.502369] usb 1-2: SerialNumber: O9-9999999999999999
[ 1079.687043] usb 1-2: can't set config #1, error -110

And then the device is unusable. I have to put it back to FX2 bootloader mode, erase the EEPROM and start again.

What kind of debug tools do you use for the FX2?

[DurandA]

After some more debug, the USB errors appear as I get closer to the memory limit of the 8051.

It is strange, because I do not see overlaps in build/glasgow.mem:

Other memory:
   Name             Start    End      Size     Max     
   ---------------- -------- -------- -------- --------
   PAGED EXT. RAM   0x3e00   0x3ecc     205      256   
   EXTERNAL RAM     0x3ecd   0x3f1a      78      448   
   ROM/EPROM/FLASH  0x0000   0x3958   14617    15872   

I know that struct glasgow_config is located at __xdata __at(0x4000 - CONF_SIZE) where CONF_SIZE is configured to 0x0040 but 0x4000-0x40 > 0x3f1a.

Commenting out a few unused functions solves the USB issues. It could be a memory alignment issue as well.

I'm not sure if I should close this issue as it could happen if the current firmware is expanded. Let me know what you think.

[whitequark]

I agree that this is a potential issue for expanding the current firmware.

[DurandA]

I started investigating if this could be a config overwriting due to an overlapping EEPROM page.

I enabled https://github.com/GlasgowEmbedded/glasgow/blob/99b81aac282f02ec3f3cda9566d929e30f1ecef4/software/glasgow/device/hardware.py#L310-L311 and https://github.com/GlasgowEmbedded/glasgow/blob/99b81aac282f02ec3f3cda9566d929e30f1ecef4/software/glasgow/device/hardware.py#L325-L326

What surprises me is that the configuration is read at address 0 of the EEPROM while I thought it was located at 0x3fc0:

$ glasgow flash --firmware firmware/glasgow.ihex
I: g.cli: reading device configuration
reading EEPROM chip 0 range 0000-004b
I: g.cli: device has serial C1-20210406T021410Z

Is the memory mapping reversed?

[whitequark]

No. Take a look at the section of the FX2 manual that describes how loads its firmware from EEPROM. In short, there's a series of length-address-value chunks at the start of the EEPROM, so the EEPROM addresses and the XRAM addresses have no correspondence.

[whitequark]

At a glance, this issue seems to have been caused by the code area becoming too large and overflowing into something important. (The atecc.c file is pretty chonky.)

In any case this is not an actionable issue for the Glasgow project.