Suggest changing compression method and dd...

[ryantig]

Two things I would suggest changing

1. Compression improvement... bzip2 compresses OK, but not great, and not fast. For speed, lz4 -9 is great, but doesn't compress as well...
2. dd image writing method. 2a. You show decompressing, then using dd to write out the image. This is /slower/ than decompressing to stdout piped to the dd. 2b. Picking a larger block size doesn't really buy you anything beyond inflated fake buffered numbers and maybe showing complete before data is actually synced

What I would suggest on #1... Since many people download the output, it's worth taking a bit longer to compress better, using xz -8 ref. Your 2017-10-08 image as bz2 is 220M, lz4 -9 is 264M, xz -6 is 167M, xz -8 is 163M. As a bonus, decompression of xz is faster than bzip2...

What I would suggest on #2a. decompress to the dd, it saves a step and it means fewer reads from your disk are needed to write the image out... What I would suggest on #2b is a block size that fits well within /sys/block//queue/max_sectors_kb To simplify vs checking, 64k is typically going to fit well, and be a good fit vs the flash device erase/page size. To help protect against reporting complete while writes are still taking place, I suggest a couple output flags, and to make sure the input pipe works well, I'll suggest a iflag as well. I also like to know how things are progressing, so I'll suggest a status argument as well, if the OS supports it (debian does)...

2 combined example...

xzcat 2017-10-08-raspberry-pi-3-buster-PREVIEW.img.xz | dd of=/dev/sdb bs=64k oflag=dsync status=progress

• xzcat: decompress an XZ file to stdout. Could use xz -dc the same way
• 2017-10-08-raspberry-pi-3-buster-PREVIEW.img.xz: Compressed image file name
• |: pipe to redirect the stdout decompressed stream
• dd: to process the data movement from the image file to disk target
• of: output file
• /dev/sdb: the device we're going to write over (make sure to select correctly) in this case /dev/sdb
• bs: block size, we're using 64k above.
• oflag: output flags
• dsync: sync data on writes. This means when the dd completes, the data is actually committed to the target device. It also means performance indications will be more accurate.
• status=progress: display performance information as the dd runs, and when it completes.

Much thanks for the work on the rp3 tools and images - I really appreciate it!

[stapelberg]

Thanks for the suggestion and detailed explanation. I’ve switched to xz and updated the instructions as you recommended.