Support arbitrarily sized initrd

[ali1234]

My initrd is 28MB and I get a message "Failed to write complete file to USB device". Smaller initrds transfer okay. (But doesn't boot due to #13.)

The 28MB initrd works fine from SD card or using netboot, so I don't see why it should not work with usbboot.

[ED6E0F17]

I have a patch for larger file sizes, https://github.com/ED6E0F17/usbboot/commit/c5aa4ff99f7f688e86d155f5027d8896ca706243 , but it is hanging during the next transfer:

Received message GetFileSize: kernel.img
File size = 6563704 bytes
Received message ReadFile: kernel.img
File read: kernel.img
libusb_bulk_transfer sent 6563704 bytes; returned 0
Received message GetFileSize: bigramdisk.img
File size = 45263708 bytes
Received message ReadFile: bigramdisk.img
File read: bigramdisk.img
libusb_bulk_transfer sent 45263708 bytes; returned 0
Received message GetFileSize: bcm2708-rpi-b-plus.dtb
File size = 15440 bytes

and it stops there

[ghollingworth]

I'm still working on this... I've found some problems that I've been fixing, but USB analyser is playing up and making it difficult to understand what is going wrong!

[ghollingworth]

Finally got to the point where I've been able to fix a couple of things, one of which was a lock up with one of the longer files, not sure whether this is the same problem you are seeing because I wasn't adjusting the file sizes but should have something for you to test later this week

Gordon

[ali1234]

This still doesn't work with git master.

al@al-desktop:~/rpi-ramdisk/boot-raspbian$ sudo rpiboot -d .
Waiting for BCM2835/6/7
Sending bootcode.bin
Successful read 4 bytes 
Waiting for BCM2835/6/7
Second stage boot server
File read: config.txt
File read: start.elf
File read: fixup.dat
File read: config.txt
File read: config.txt
File read: cmdline.txt
File read: kernel.img
File read: initrd
Failed to write complete file to USB device

[ali1234]

Through trial and error I found that the maximum sized initrd which will work is somewhere between 16744192 bytes and 16744244 bytes.

[ali1234]

Verbose log:

Received message GetFileSize: initrd
File size = 16744320 bytes
Received message ReadFile: initrd
File read: initrd
libusb: error [submit_bulk_transfer] submiturb failed error -1 errno=12
libusb_bulk_transfer returned -1
Failed to write complete file to USB device (0 bytes written)

12 is ENOMEM.

[ali1234]

Looking inside the linux kernel at the submiturb stuff I see this:

/* Limit on the total amount of memory we can allocate for transfers */
static unsigned usbfs_memory_mb = 16;
module_param(usbfs_memory_mb, uint, 0644);
MODULE_PARM_DESC(usbfs_memory_mb,
                "maximum MB allowed for usbfs buffers (0 = no limit)");

This seems to suggest that there is a default limit of 16MiB for transfer buffers.

Checking /sys/module/usbcore/parameters/usbfs_memory_mb shows it is indeed set to 16 on Ubuntu 16.04. After writing 32 into it, I was able to transfer an initrd which was previously too large.

After writing 128 I was able to transfer my full Raspbian initrd, which is 72470431 bytes. However, rpiboot hung after the transfer finished. It didn't continue on with transfering device trees, and no output on the serial console.

Trimming the initrd down to 52975398 bytes had the same result.

After extreme trimming down to 30721773 bytes, the initrd was able to boot correctly.

[dglaude]

So currently echo 32 | sudo tee /sys/module/usbcore/parameters/usbfs_memory_mb is a fix for some size, but it is very annoying to have to add this in end-user instruction. Especially if they are using usbbootgui and they have to go back to CLI (after a first failed usbboot).

@ghollingworth are you working on a fix, or would you like to hide the problem in usbbootgui to automatically detect that the initrd file selected by the user is big and usbboot require a little help to work?

[ghollingworth]

Looks to me from a quick search of google that this is likely a problem with using libusb_bulk_transfer since that's occurring with some other devices as well... I'd suggest trying replacing with an asynchronous interface rather than the synchronous one (I assume each transmission block is being stored in usbfs for transmission but async should enable it to be sent in small blocks)

https://falsinsoft.blogspot.co.uk/2015/02/asynchronous-bulk-transfer-using-libusb.html

If anyone is up for having a play with usbboot

[ali1234]

The maximum size of a USB2 packet on a bulk endpoint is 512 bytes... if you send anything larger than that (sync OR async) it is stored in a buffer inside the kernel, and the kernel fragments it into 512 byte chunks before sending.

The maximum size of that buffer is controlled by the "usbfs_memory_mb" module parameter which defaults to 16MB.

All you have to do, in order to not exceed the size of that buffer, is send the file with multiple calls to libusb_bulk_transfer(), like was suggested earlier in this thread, and also here https://github.com/raspberrypi/tools/pull/20

Sending using the async API does not make any difference - the data still ends up in the same buffer inside the kernel, so you must still split it in to chunks smaller than usbfs_memory_mb before you write.

[pelwell]

Looking back through the thread I see that @oandrew has a patch in his repo that claims to fix the issue, but he didn't create a PR for it. @ali1234 Does this patch work for you?

[ali1234]

@pelwell he did send a pull request. It is here: https://github.com/raspberrypi/usbboot/pull/18

Neither that patch nor https://github.com/ED6E0F17/usbboot/commit/c5aa4ff99f7f688e86d155f5027d8896ca706243 which is another sync version work currently. They all successfully transfer the file, but then the device hangs.

I have not tried rebasing the patches on top of current master. That is probably why none of them work.

[ali1234]

Today I tested rebasing both available patches against master. Both work.

I have put the rebased patches into a fork of this repo. Each is in it's own branch, here:

https://github.com/ali1234/usbboot/tree/hexameron (from https://github.com/ED6E0F17/usbboot/commit/c5aa4ff99f7f688e86d155f5027d8896ca706243)

Synchronous version, ultimately based on https://github.com/raspberrypi/tools/pull/20. Works. Applies clean to master. No compile warnings. Takes 53 seconds to load firmware.

https://github.com/ali1234/usbboot/tree/oandrew (from https://github.com/oandrew/usbboot/commit/ae76ffdb43521f796b508229d01fd12280c41eb1)

Asynchronous version. Also works. Does not apply clean to master. Generates compiler warnings. Makes some changes in Makefile that I have not evaluated. Takes 54 seconds to load firmware.

I strongly recommend using a synchronous implementation as the code is far simpler. There also doesn't seem to be any point in doing async writes, because rpiboot has nothing else to do while waiting for the writes to complete. The synchonous version seems to be slightly faster, but the performance difference is within the margin of error.

Also note that both patches use magic values for the maximum transfer size. The true maximum size is indirectly controlled by /sys/module/usbcore/parameters/usbfs_memory_mb - this value sets the total size of all buffers allocated by usbfs (in megabytes), so it would be wise to stay well under this limit. The default value is 16MB.

The synchonous patch uses a block size of 4MB. This is quite large. The asynchonous patch uses a 16KB block size. Given the way the memory limit works, I would recommend using 16KB blocks. Larger transfers are always guaranteed to be split into packets of 16KB or less inside the kernel.

I have created a version of the synchonous patch which uses 16KB blocks. This required either removing or greatly increasing the abort count. Since the loop will always about after a single error, the abort count seems to serve no purpose. So I removed it. In testing I found that using 16KB blocks instead of 4MB blocks had no observable effect on transfer speed. The code is here:

https://github.com/ali1234/usbboot/tree/smallpacket

This is the version I would recommend people actually use.

[pelwell]

This is now a PR: https://github.com/raspberrypi/usbboot/pull/26

If accepted we can squash the changes and come up with a suitable commit message.

[ali1234]

Thanks. Note that there is still another issue preventing transfer of files larger than about 32MB. I suspect it is overrunning a buffer inside bootcode.bin, because the file will transfer successfully but then the device will just freeze up.

[ED6E0F17]

Resolved by : https://github.com/raspberrypi/usbboot/commit/c29a36a4245432702c6242d625c5408cf634b1f7

The code is designed to loop and retry if there is a timeout, but the repeated attempts also fail, and there is no error reporting in that section of the code to indicate what was going wrong..

[mi-hol]

Would "Resolved by : c29a36a" not trigger closure of this issue?