Flash Process slows to a crawl around 95% on big gzip images

[petealbertson]

• Etcher version: 1.0.0-beta.19
• Operating system and architecture: MacOS Sierra 10.12.3 MacBook Air
• Do you see any meaningful error information on DevTools? No errors in the console. Only lots of progress logs.

I'm attempting to flash an image of Android 6 to a 32BG Sandisk microSD card and for the second consecutive attempt, it slows to a crawl around 95%. It's now at a write speed of 0.64MB/s and slowing. Is there an abort protocol?

[jhermsmeier]

Hey @petealbertson, can you tell us which exact image you're using, and (if not obvious) where to get it, so that we can have an in depth look at this?

[lurch]

Sounds similar to #638 ?

[jhermsmeier]

@lurch Not sure, might be, but Android images usually aren't that big, and come in .zips.

[jviotti]

Yeah, the gz issue was the first thing that came to my mind as well. Let's see what image the user was flashing.

[petealbertson]

Yes, it is an Android image from https://www.pine64.org/?page_id=1986

[jhermsmeier]

Which one of those exactly did you use?

[petealbertson]

The 32gb version direct from Pine (the first link under 32gb).

[jhermsmeier]

Okay, so those images are gzipped, not .zips, plus their contents are all over 4GB, which means #638 is most likely the culprit.

A workaround would be to extract the .img before flashing it, if you have enough disk space (the pine image will decompress to ~32GB).

[jviotti]

the pine image will decompress to ~32GB

Wow, that's the record so far :)

@jhermsmeier Did you have a chance to investigate the gzip issue in more detail? I'd love to hear your thoughts about what can we do to fix it.

[lurch]

That's insane, the Pine developers have obviously never heard of partition resizing and filesystem expansion ;-) (note that each of the different images compresses down to a similar size, indicating that they all have almost identical contents)

[lurch]

....oh, except of course these are Android images, and Android probably doesn't have the tools to resize partitions like Linux does. Whoops :confused:

[lurch]

Thinking about it, even if we did add code to 'guesstimate' that this image is actually 4GB larger than it claims to be, that still wouldn't help us here, because we'd still 'guess' at the image being less than 8GB in total (it's massively compressed because it's mainly empty space), and so Etcher would still appear to "slow down near the end" when it's written 8GB but the image it's writing is actually 32GB :-(

Maybe it'd be worth getting in touch with the Pine64 team and suggesting (given that they're recommending Etcher) that they use .xz or .zip compression instead?

[jviotti]

The estimation is only used to prevent the user from selecting a too small drive, but its not used for the actual writing at all. The writer calculates the percentage and speed based on the compressed size. I investigated the issue a bit (although not in depth), and the issue seems to be that gzip gets uncompressed very fast (gzip support comes bundled with node), and therefore the decompression happens way faster than the actual writing, so the percentage reaches the ~90% range very quickly, but then its stuck writing to the card.

[lurch]

Ahhh. Could you use the "back-pressure" thing, so that the image-decompression gets paused if it has output more data than has been written to disk?

Fun stats: android-ver6.0-20170112-pine64-32GB.img.gz has a raw size of 815284283 bytes (i.e. 778MiB). gunzip -l reports it has an uncompressed size of 843579392 bytes (i.e. 805MiB). gunzip -c android-ver6.0-20170112-pine64-32GB.img.gz | wc -c reports the actual image size as 30908350464 bytes (i.e. 29477MiB or 29GiB). The latter command takes just under 3 minutes to run on my i5 laptop.

[jhermsmeier]

Yeah, stream backpressure should make the gunzip stream only read more once the writing stream requests more data. But that could be put out of effect, if the highWaterMarks are too high, or if put into flowing mode (which happends when something does .resume() or .on('data', fn) on a stream).

[jhermsmeier]

Having a gzipped imaged padded with 20+ GB of zeros is pretty unusual (and also ineffective), but apparently it works.

Android probably doesn't have the tools to resize partitions like Linux does. Whoops

I don't think that there'd be any fancy filesystems underneath – after all Android is based on the Linux kernel – so in theory offline partition & fs expansion should be possible after flashing.

[lurch]

For the sake of comparison (in case it's useful) here's the info from all the Pine64 Android images on that page:

Filename	Compressed size	Uncompressed size	Size reported by gunzip -l	Time taken to decompress
android-ver6.0-20170112-pine64-8GB.img.gz	724252876 (~691MiB)	7780433920 (7420MiB)	3485466624 (3324MiB)	0m48s
android-ver6.0-20170112-pine64-16GB.img.gz	792061090 (~755MiB)	15523119104 (14804MiB)	2638217216 (2516MiB)	1m31s
android-ver6.0-20170112-pine64-32GB.img.gz	815284283 (~778MiB)	30908350464 (29476.5MiB)	843579392 (804.5MiB)	2m55s
android-ver6.0-20170112-pine64-64GB.img.gz	774115873 (~738MiB)	63282610176 (60351MiB)	3153068032 (3007MiB)	5m57s

(and all the MD5 checksums listed on that page are wrong, which isn't very helpful! (but then they do claim MD5 (Zip file) whereas the files you actually download are .gz :confounded: ) )

The 'odd' uncompressed sizes makes me suspect that these are simply images that have been created by dd-ing back from an actual SD card (which is going to be unfortunate for people who have SD cards slightly smaller than those used by Pine64).

[lurch]

Hmmm, I know absolutely nothing about Android or the Pine64, but their partitioning looks a little 'strange' :-S

$ fdisk -l android-ver6.0-20170112-pine64-8GB.img
omitting empty partition (18)

Disk android-ver6.0-20170112-pine64-8GB.img: 7780 MB, 7780433920 bytes
1 heads, 63 sectors/track, 241208 cylinders, total 15196160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

                                 Device Boot      Start         End      Blocks   Id  System
android-ver6.0-20170112-pine64-8GB.img1         6430720    15194110     4381695+   b  W95 FAT32
android-ver6.0-20170112-pine64-8GB.img2   *       73728      139263       32768    6  FAT16
android-ver6.0-20170112-pine64-8GB.img3               1     6430720     3215360   85  Linux extended
android-ver6.0-20170112-pine64-8GB.img5          139264      172031       16384   83  Linux
android-ver6.0-20170112-pine64-8GB.img6          172032      204799       16384   83  Linux
android-ver6.0-20170112-pine64-8GB.img7          204800     4399103     2097152   83  Linux
android-ver6.0-20170112-pine64-8GB.img8         4399104     4464639       32768   83  Linux
android-ver6.0-20170112-pine64-8GB.img9         4464640     4497407       16384   83  Linux
android-ver6.0-20170112-pine64-8GB.img10        4497408     4562943       32768   83  Linux
android-ver6.0-20170112-pine64-8GB.img11        4562944     6135807      786432   83  Linux
android-ver6.0-20170112-pine64-8GB.img12        6135808     6168575       16384   83  Linux
android-ver6.0-20170112-pine64-8GB.img13        6168576     6201343       16384   83  Linux
android-ver6.0-20170112-pine64-8GB.img14        6201344     6202367         512   83  Linux
android-ver6.0-20170112-pine64-8GB.img15        6202368     6234111       15872   83  Linux
android-ver6.0-20170112-pine64-8GB.img16        6234112     6397951       81920   83  Linux
android-ver6.0-20170112-pine64-8GB.img17        6397952     6430719       16384   83  Linux

Partition table entries are not in disk order

[jviotti]

I seem to be able to reproduce this issue with xz on a not very high end Windows laptop.