search

balena-io / etcher

Flash OS images to SD cards & USB drives, safely and easily.
https://etcher.io/
Apache License 2.0
29.86k stars 2.11k forks source link

LZ4 compatibility #2048

Closed AlexisTM closed 6 years ago

AlexisTM commented 6 years ago

What do you think of implementing LZ4 decompression? Indeed, this is awesomely fast (0.6GB/s compression, 3.2GB/s decompression) algorithm at the price of a lesser compression ratio (~2.0 for data). Yet, for images, there is a lot of empty space which is well compressed. (32GB SD card dump images to 6GB).

lurch commented 6 years ago

Looks like there's already an LZ4 NodeJS module we could use https://github.com/pierrec/node-lz4

@AlexisTM In order to judge demand for this, are you aware of any distros / OSes that distribute their images in LZ4 format? I believe most distros probably tend to 'optimise' for greater compression ratio (i.e. smaller download sizes) over faster decompression rates? Also, given that Etcher is typically used to write to SD cards, the physical write-speeds are probably more of a bottleneck than the decompression speed? :wink:

This suggestion might become more useful once Etcher has the ability to create (compressed) backups from SD cards? #266

Shou commented 6 years ago

@lurch I guess it's kind of a chicken and egg situation, maybe nobody's distributing LZ4-compressed images due to lack of software support, so when we implement it we might see subsequent adoption. Considering LZ4 HC ranks among zlib in terms of transfer while still decompressing much faster, the user experience is probably overall better with LZ4.

jhermsmeier commented 6 years ago

the physical write-speeds are probably more of a bottleneck than the decompression speed?

Yup, unless you're flashing entire SSDs for fun & profit the compression scheme doesn't make much of a difference (esp. not a noticeable one) – except for bzip2 maybe, that one's noticeably slower.

AlexisTM commented 6 years ago

@lurch I do use it personally. I had to take a backup, then store it online.

I started a gz compression and after 30 minutes it was still working... I stopped it; I then used LZ4 and in 2 minutes it was finished. Yet, as you say it; I never saw any distro distributing in LZ4.