Use "-no-fragments" mksquashfs

[develar]

Please see https://forum.snapcraft.io/t/proposal-to-disable-squashfs-fragments-in-snaps/3103/8

As we need to produce consistent results (important for differential updates).

[TheAssassin]

Seems like a sane option. This is part of #625. Actually, we have a test checking whether two runs of appimagetool produce two exactly same AppImages (checked by calculating and comparing the files' checksums) on Travis CI. There, appimagetool is run on a multi-processor system with 2 CPU cores, and I assume the builds are run with 2 threads. I wonder whether those checks are reliable now, though. They seem to work on my local machine as well.

If we consider using -nofrag, it also seems like we have to reconsider the block size we use by default then. 128 kiB is a bit much for a single file. Some users in the thread have experienced some notable increases of the file size.

[probonopd]

If we consider using -nofrag, it also seems like we have to reconsider the block size we use by default then.

Why?

[TheAssassin]

@probonopd the reference to the snap discussion explains what -nofrag does: It causes files to not be packed together but to be in separate blocks.

Therefore:

128 kiB is a bit much for a single file. Some users in the thread have experienced some notable increases of the file size.

[probonopd]

Uh. So you are saying we are getting larger AppImages just to have them reproducible? How large is this tradeoff in real life?

[TheAssassin]

See my comment https://github.com/AppImage/AppImageKit/issues/695#issuecomment-370703609. I don't have any data on it. I just assumed that since we also have a lot of small files, there is a slight increase of size to be expected. In the example in the snap forum, the size of some snap went from 500 to 511MiB. That's an increase of ~2%. That'd be acceptable in my opinion.

[probonopd]

2% seems acceptable to me if this means that binary delta downloads will be smaller as a result.

[TheAssassin]

Well, only if we are sure the order of the blocks is reproducible, no matter whether threads are activated or not (with a single thread, the order should be reproducible anyway, as long as they didn't try to use some randomized algorithm...). But I could imagine it'll help reduce the download sizes.

I think we should have someone perform some comparisons before making changes. Without any data, we can only guess.

[develar]

electron-builder uses -no-fragments. Size is not changed (~51.7 sample app).

[azubieta]

We could also use this to mark the builds: https://fedoraproject.org/wiki/Releases/FeatureBuildId

[TheAssassin]

@azubieta how is this related to -no-fragments?

[azubieta]

Sorry for the spam I just wasn't able to find were reproducible builds were discussed. This is used to tag a build whit an unique id.

El 18 may. 2018 11:29 PM, "TheAssassin" notifications@github.com escribió:

@azubieta https://github.com/azubieta how is this related to -no-fragments ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AppImage/AppImageKit/issues/695#issuecomment-390377854, or mute the thread https://github.com/notifications/unsubscribe-auth/ABFdrip2ln6R9ijJRhjWEPJPBpYSDQYpks5tz5-9gaJpZM4SeRzR .

[probonopd]

According to https://forum.snapcraft.io/t/proposal-to-disable-squashfs-fragments-in-snaps/3103/8, -no-fragments does increase the size of the squashfs files, while patching mksquashfs to not use threads would achieve a similar result without increasing the size (at the cost of a slower compression). I'd go for the latter option, since size matters more than compression speed in our use case.

[TheAssassin]

@probonopd there's a secondary aspect that'd be more efficent with -no-fragments: updating. With this flag, the files will probably end up "block aligned" in the squashfs image, i.e., for every new file a new block is created. That could make delta updating a lot easier. For example, if a big file is changed to which a big file is appended, and the second file starts in the last block of the first one, it is likely that the entire blocks of the second file will change, maybe even following files. Therefore, the 2% increase that I mentioned before are worth it, given that my assumption is correct.