Update start.elf for customized Raspberry Pis

[acostach]

Hi @ghollingworth , not sure if you still work with this, would it be possible to have start.elf updated to put the same customized Pis in msd mode as you've done in f4e3f0f9a3c64d846ba53ec3367e33a4f9a7d051 ? I've replaced start.elf with msd.elf from your commit in the HEAD revision and it puts the device in mass storage mode, however with the current latest start.elf it doesn't work.

Thank you and please let me know if I can test some commit or branch.

[pelwell]

That change just told the firmware that custom boards exist and what their properties are. That information is still in the current firmware, so something else must have changed to break it.

The commits page (https://github.com/raspberrypi/usbboot/commits/master) shows all the releases since https://github.com/raspberrypi/usbboot/commit/f4e3f0f9a3c64d846ba53ec3367e33a4f9a7d051. There are a lot, and testing them all would take a long time. Fortunately git can help:

$ git bisect start
$ git bisect good f4e3f0f
$ git bisect bad origin/master

git chooses a point halfway between the good and bad pointers.

$ make                 # Just to be sure
$ ./rpiboot -d msd

If it works:

$ git bisect good

otherwise:

$ git bisect bad

Repeat from make above until you find the commit that caused the problem. Make a note of it and be sure to include it in a comment here. Then to revert to normal operation:

$ git bisect reset

[acostach]

Thank you for the detailed steps @pelwell , the first commit that breaks this is:

d7b97b343fd09c68792f4f9a8975af261efda1a2 is the first bad commit
commit d7b97b343fd09c68792f4f9a8975af261efda1a2
Author: ghollingworth <gordon@raspberrypi.org>
Date:   Tue Oct 25 16:15:07 2016 +0100

    Changes to add filesystem boot mode

:100755 000000 3e047ed96dab4c6d7c9c850a2665a1ce53a1f4b1 0000000000000000000000000000000000000000 D  buildroot.elf
:100644 000000 1fbb367bcbe9bde7f0c8f4357f480632a98c10cd 0000000000000000000000000000000000000000 D  buildroot.patch
:100755 100755 1b4e0428c19ac0e6528c45d929e0c707eef98d44 7c25c8f7acad7edc13c8c07722c27b5b7bb54dfc M  main.c
:100755 000000 fd7a0fd71995f655f1f91ab676641d4f932afae1 0000000000000000000000000000000000000000 D  msd.elf
:000000 040000 0000000000000000000000000000000000000000 b75cccb39612a9ffbc1d9a4efa1a9fa43edc0050 A  msd

[pelwell]

Thanks - we have some ideas of what to try first.

[acostach]

Thank you, please let me know when I can test them.

[pelwell]

The main difference between those two commits is that msd.elf was moved and renamed to msd/start.elf - the files themselves are exactly the same - with rpiboot being updated accordingly.

Please run sudo ./rpiboot -v -d msd (note the -v flag for verbose mode) on a working and non-working system so we can see which files are being loaded in each case.

[acostach]

Thanks @pelwell , I've added logs from 4 cases here: https://gist.github.com/acostach/523d2ddc5f16d3a6e6d13fb4d6225a10

Indeed, I've now seen that msd.elf was renamed, the failure in d7b97b343fd09c68792f4f9a8975af261efda1a2 is most likely unrelated. Please let me know what I should test further.

[pelwell]

Am I right in thinking that the only visible difference between the two cases (apart from the start.elf file size) is that in the failure case the MSD devices never appear?

Does your custom device have a jumper or similar to disable the EMMC/SD card access for rpibooting? If so, which GPIO is used to drive the override to re-enable access to the storage medium for MSD?

[acostach]

@pelwell indeed, there's no msd device created in the failure case. No, the pi does not use any jumper or gpio, I just plug the micro-usb program port in the host PC, then plug in the micro-usb power and that's enough to start running usbboot.

[ghollingworth]

Which GPIO is used to re-enable the eMMC ?

[pelwell]

The boot ROM will always try booting from SD/EMMC before USB, so rpiboot relies on something preventing the device from accessing the local storage. For some applications, on say a Zero, removing the SD card is sufficient, but for MSD use it has to be reinserted while the image is being downloaded - the window is several seconds long so the timing isn't difficult.

What is stopping the custom Pi booting from the SD/EMMC?

[acostach]

To stop it from booting from the emmc it's enough to plug in the micro usb debug port to the host PC, and then power the device trough the power micro usb port. That's all.

I could only find this in the custom dtblob:

pin_define@EMMC_ENABLE {
   type = "external";
   number = <6>;
};

Also, there's no sd-card on this type of device @pelwell

[pelwell]

There is a change to the msd application that might be responsible for the breakage. If I'm right, fb86716935f2e820333b037a2ff93a338ad9b695 should contain the last good version and c3ee230ef4a8f247b7b7d406e59beec8c895de60 the first broken one. Can you confirm that?

[acostach]

Thanks @pelwell , I've tried fb86716 and it unfortunately it still doesn't show up as msd. The logs are the following:

sudo ./rpiboot -d msd/ -v > fb86716.log 2>&1
Waiting for BCM2835/6/7
Device located successfully
Initialised device correctly
Found serial number 0
Sending bootcode.bin
libusb_bulk_transfer returned 0
Writing 51992 bytes
libusb_bulk_transfer returned 0
Successful read 4 bytes 
Waiting for BCM2835/6/7
Device located successfully
libusb: error [udev_hotplug_event] ignoring udev action bind
Failed to open the requested device
Device located successfully
Initialised device correctly
Found serial number 1
Second stage boot server
Received message GetFileSize: autoboot.txt
Cannot open file autoboot.txt
Received message GetFileSize: config.txt
Cannot open file config.txt
Received message GetFileSize: recovery.elf
Cannot open file recovery.elf
Received message GetFileSize: start.elf
File size = 433616 bytes
Received message ReadFile: start.elf
File read: start.elf
libusb_bulk_transfer returned 0
Received message GetFileSize: fixup.dat
Cannot open file fixup.dat
Second stage boot server done

After that I replaced msd/start.elf with the on known to work, did not change anything else, make clean && make:

usbboot$ git diff msd/
diff --git a/msd/start.elf b/msd/start.elf
index aedee32..fd7a0fd 100755
Binary files a/msd/start.elf and b/msd/start.elf differ
usbboot$ md5sum msd/start.elf 
735af16c0652dfdff701d5b794e85cae  msd/start.elf

And it worked fine, msd was mounted:

usbboot$ cat fb86716_old_msd_elf.log
Waiting for BCM2835/6/7
Device located successfully
Initialised device correctly
Found serial number 0
Sending bootcode.bin
libusb_bulk_transfer returned 0
Writing 51992 bytes
libusb_bulk_transfer returned 0
Successful read 4 bytes 
Waiting for BCM2835/6/7
Device located successfully
libusb: error [udev_hotplug_event] ignoring udev action bind
Failed to open the requested device
Device located successfully
Initialised device correctly
Found serial number 1
Second stage boot server
Received message GetFileSize: autoboot.txt
Cannot open file autoboot.txt
Received message GetFileSize: config.txt
Cannot open file config.txt
Received message GetFileSize: recovery.elf
Cannot open file recovery.elf
Received message GetFileSize: start.elf
File size = 428476 bytes
Received message ReadFile: start.elf
File read: start.elf
libusb_bulk_transfer returned 0
Received message GetFileSize: fixup.dat
Cannot open file fixup.dat
Second stage boot server done
libusb: error [udev_hotplug_event] ignoring udev action bind

I've aslo tried the second commit c3ee230ef4a8f247b7b7d406e59beec8c895de60, no changes, only make clean && make: Doesn't work:

sudo ./rpiboot -d msd/ -v
Waiting for BCM2835/6/7
Device located successfully
libusb: error [udev_hotplug_event] ignoring udev action bind
Initialised device correctly
Found serial number 0
Sending bootcode.bin
libusb_bulk_transfer returned 0
Writing 51992 bytes
libusb_bulk_transfer returned 0
Successful read 4 bytes 
Waiting for BCM2835/6/7
Device located successfully
libusb: error [udev_hotplug_event] ignoring udev action bind
Failed to open the requested device
Device located successfully
Initialised device correctly
Found serial number 1
Second stage boot server
Received message GetFileSize: autoboot.txt
Cannot open file autoboot.txt
Received message GetFileSize: config.txt
Cannot open file config.txt
Received message GetFileSize: recovery.elf
Cannot open file recovery.elf
Received message GetFileSize: start.elf
File size = 438936 bytes
Received message ReadFile: start.elf
File read: start.elf
libusb_bulk_transfer returned 0
Received message GetFileSize: fixup.dat
Cannot open file fixup.dat
Second stage boot server done

Then I replaced again msd/start.elf with the old known to work one with MD5 735af16c0652dfdff701d5b794e85cae , no other changes, make clean && make, same commit that failed but with the old start.elf and it worked:

commit c3ee230ef4a8f247b7b7d406e59beec8c895de60 (HEAD)
Author: Gordon Hollingworth <gordon@raspberrypi.org>
Date:   Thu Apr 5 17:03:37 2018 +0100

    Change USB device settings to better work with all devices

sudo ./rpiboot -d msd/ -v
Waiting for BCM2835/6/7
Device located successfully
Initialised device correctly
Found serial number 0
Sending bootcode.bin
libusb_bulk_transfer returned 0
Writing 51992 bytes
libusb_bulk_transfer returned 0
Successful read 4 bytes 
Waiting for BCM2835/6/7
Device located successfully
libusb: error [udev_hotplug_event] ignoring udev action bind
Failed to open the requested device
Device located successfully
Initialised device correctly
Found serial number 1
Second stage boot server
Received message GetFileSize: autoboot.txt
Cannot open file autoboot.txt
Received message GetFileSize: config.txt
Cannot open file config.txt
Received message GetFileSize: recovery.elf
Cannot open file recovery.elf
Received message GetFileSize: start.elf
File size = 428476 bytes
Received message ReadFile: start.elf
File read: start.elf
libusb_bulk_transfer returned 0
Received message GetFileSize: fixup.dat
Cannot open file fixup.dat
Second stage boot server done
libusb: error [udev_hotplug_event] ignoring udev action bind

[acostach]

I could also go trough the commits that changed start.elf and test them on some variant that's known to work, like f4e3f0f, if that made sense @pelwell

[pelwell]

Yes - that would be helpful.

[acostach]

Sure @pelwell , I checked and this is what I've observed:

First commit that changes start.elf after the revision that worked is

commit ec9155c9784ee9bd91c9179a1fd3d9ca8f963cd3
Author: ghollingworth <gordon@raspberrypi.org>
Date:   Sat May 13 10:28:40 2017 +0100

    Fix problem with enabling MSD on some CM / CM3 devices
---
 msd/start.elf | Bin 428476 -> 433448 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/msd/start.elf b/msd/start.elf
index fd7a0fd..192a49c 100755
Binary files a/msd/start.elf and b/msd/start.elf differ

and it is also the first msd.elf/start.elf that stopped working.

Verbose log: https://gist.github.com/acostach/a4aaf216ec964750c14c63b1843bde80#file-f4e3f0f9a3c64d8_with_start_elf_from_ec9155c9784e-log

So the start.elf from it and commit f4e3f0f9a3c64d (which works by default) the device is not put in MSD mode. However, if I put msd.elf from f4e3f0f9a3c64d in this commit ec9155c9784e, the device is put in MSD mode and the only difference is that ./rpiboot never returns:

sudo ./rpiboot 
Waiting for BCM2835/6/7
Sending bootcode.bin
Successful read 4 bytes 
Waiting for BCM2835/6/7
Second stage boot server
File read: start.elf

Next commit that modifies start.elf is

commit 50fc0f49eaae7cd6dbe8e85d7b7a8b745966aee3
Author: ghollingworth <gordon@raspberrypi.org>
Date:   Fri May 26 11:38:12 2017 +0100

    Fix Pi Zero and some CM3 MSD boot cases, also secure boot support
---
 msd/start.elf | Bin 433448 -> 433524 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/msd/start.elf b/msd/start.elf
index 192a49c..cbc0eb9 100755
Binary files a/msd/start.elf and b/msd/start.elf differ

its start.elf doesn't work either with the known to work revision f4e3f0f9a3c64d8.

Next commit that adds changes to start.elf is:

commit c0af212f2fee443af8371a64f5f9bced6d83d83c (HEAD -> lalala)
Author: ghollingworth <gordon@raspberrypi.org>
Date:   Fri Jun 2 09:56:03 2017 +0100

    Add configurable eMMC re-enabling value

And the start.elf from this one unfortunately doesn't work either.

[pelwell]

I've run out of time this week - continue using the old version for now and I'll try to get the magic patch from Mr Hollingworth.

[acostach]

Thanks @pelwell , please let me know when I can test something. Unfortunately integrating the old version is not possible since it will most likely cause trouble with flashing other types of devices, at least by looking at 50fc0f49eaae7cd6dbe8e85d7b7a8b745966aee3

[pelwell]

I've found the problem, and should be able to give you at least a trial MSD start.elf today.

[pelwell]

I have an updated image that appears as an MSD as before, but I'm concerned that it doesn't know about the EMMC_ENABLE pin used by your custom board.

Are you currently patching the downloaded msd/start.elf to specify the custom EMMC_ENABLE pin? There's a magic byte sequence in the .elf file (0xbe 0xba 0xe0 0xac) that should be replaced in your case with the GPIO number, (0x86 0x00 0x00 0x00 in your case).

[pelwell]

You can download the trial image (which you'll have to rename to start.elf) here: https://drive.google.com/file/d/1GHa0QXTafw_QpyyX1u0IOQA1Em06GO1t/view?usp=sharing

[acostach]

Thank you for the trial image @pelwell. No, we're not patching any of the binaries, we're using them as they are for all boards. Hopefully this firmware could work for all boards because the device is not distinguishable by vid/pid.

My test results with the binary you prepared are: 1) Un-patched: doesn't work 2) Patched: works fine

[pelwell]

So there is no misunderstanding, can you explain what you mean by unpatched and patched?

[acostach]

Hi @pelwell, by unpatched I mean the original start.elf you provided in g-drive, and patched is the same binary but with the original sequence 0xbe 0xba 0xe0 0xac replaced by 0x86 0x00 0x00 0x00:

00047ACC 00 00 00 00 00 00 00 00 **86 00 00 00** 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <- this put the device in msd mode

[pelwell]

Thanks - that's what I would expect it to mean, but I was thrown by your comment that "we're not patching any of the binaries".

As I said above, the issue with supporting custom boards is that the MSD start.elf has to know which GPIO to use as an EMMC_ENABLE. If they all use the same pin - GPIO 6 on the expander in your case - then a single patched binary can be used for all devices, otherwise you will need multiple versions and some way of ensuring you download the correct one. I can think of nicer ways to provide the EMMC_ENABLE value, but the problem of associating boards to pins isn't one I can solve for you.

So, is using expander GPIO 6 as EMMC_ENABLE common to all your custom boards?

[acostach]

Thanks @pelwell, this is the only type of custom board based on the Pi3 that we're dealing with, so the answer is Yes, it is common for all units of this custom device type. I don't have other custom types to check.

Regarding the other boards, I was referring mostly to the CM3, trying to ensure that this change doesn't affect them. I checked a RevPi Core3 and a BalenaFIN with a 2018 CM3+ in it and they worked fine with this new patched binary.

[pelwell]

That's great. Although I like to make sweeping, unifying changes, the risk of breaking the corner cases outweighs any benefits in this case. The trial build I gave you can become the next release, and the software patch that made it work will be merged to our internal software repo.

Make a note somewhere that any future updates from us will require the same patching process - it's not a bug fix, it's a low-tech, low-overhead communication channel from the user to the MSD application.

[acostach]

Thank you very much again for your help, looking forward to test the next release when it will become available.

[pelwell]

It's there now.

[acostach]

Implemented in the above commit.