search

AgentD / squashfs-tools-ng

A new set of tools and libraries for working with SquashFS images
Other
196 stars 30 forks source link

Possible non-determinism or format changes in some compressors #33

Closed AgentD closed 4 years ago

AgentD commented 4 years ago

In #32 @bdrung wrote:

tests/cantrbry.sh and tests/test_tar_sqfs.sh failed with sha512sum complaining about mismatched checksums.

Those tests were recently added to catch regressions in the output generated by libsquashfs or other non-deterministic/non-reproducible behaviour. The idea being, that if everything works perfectly deterministic and the same output is produced every time, the SHA-512 hashes should be the same.

I re-tested this on an OpenSuSE build machine I have access too and also got some checksums rejected. It affected all tests for zstd and lz4. Everything else worked.

Since my Fedora laptop where I generated those checksums (and where all tests succeed) has newer versions for those libraries than the OpenSuSE machine and the images produced on the later are all slightly bigger, my assumption is that libzstd and liblz4 improved their compression schemes in between those two versions. On a first glance, the change log for lz4 (1.8.0 vs 1.9.1) seems to support that assumption. Also, zstd being unstable doesn't really come as a surprise.

@bdrung: Do you get the same results or do you also have other compressors breaking as well?

bdrung commented 4 years ago

First retry gave me:

FAIL: tests/cantrbry.sh                                                                                                                                                                                                                       
=======================                                                                                                

cantrbry_gzip_4096_1.sqfs: File exists                                                                                                                                                                                                        
FAIL tests/cantrbry.sh (exit status: 1)                                                                                                                                                                                                       

FAIL: tests/test_tar_sqfs.sh                                                                                                                                                                                                                  
============================                                                                                                                                                                                                                  

tests/tar/large-mtime/pax.sqfs: File exists                                                                                                                                                                                                   
FAIL tests/test_tar_sqfs.sh (exit status: 1)

I did a cleanup:

find * -name '*.sqfs' -delete

Now I get the same result as initially. It seems that all checksums mismatch:

FAIL: tests/cantrbry.sh                    
=======================                    

cantrbry_gzip_1048576_1.sqfs: GESCHEITERT            
cantrbry_gzip_1048576_1_T.sqfs: GESCHEITERT   
cantrbry_gzip_131072_1.sqfs: GESCHEITERT
cantrbry_gzip_131072_1_T.sqfs: GESCHEITERT
cantrbry_gzip_16384_1.sqfs: GESCHEITERT
cantrbry_gzip_16384_1_T.sqfs: GESCHEITERT
cantrbry_gzip_262144_1.sqfs: GESCHEITERT
cantrbry_gzip_262144_1_T.sqfs: GESCHEITERT
cantrbry_gzip_32768_1.sqfs: GESCHEITERT
cantrbry_gzip_32768_1_T.sqfs: GESCHEITERT
cantrbry_gzip_4096_1.sqfs: GESCHEITERT 
cantrbry_gzip_4096_1_T.sqfs: GESCHEITERT 
cantrbry_gzip_524288_1.sqfs: GESCHEITERT
cantrbry_gzip_524288_1_T.sqfs: GESCHEITERT
cantrbry_gzip_65536_1.sqfs: GESCHEITERT
cantrbry_gzip_65536_1_T.sqfs: GESCHEITERT
cantrbry_gzip_8192_1.sqfs: GESCHEITERT
cantrbry_gzip_8192_1_T.sqfs: GESCHEITERT
cantrbry_lz4_1048576_1.sqfs: GESCHEITERT
cantrbry_lz4_1048576_1_T.sqfs: GESCHEITERT
cantrbry_lz4_131072_1.sqfs: GESCHEITERT
cantrbry_lz4_131072_1_T.sqfs: GESCHEITERT
cantrbry_lz4_16384_1.sqfs: GESCHEITERT
cantrbry_lz4_16384_1_T.sqfs: GESCHEITERT
cantrbry_lz4_262144_1.sqfs: GESCHEITERT
cantrbry_lz4_262144_1_T.sqfs: GESCHEITERT
cantrbry_lz4_32768_1.sqfs: GESCHEITERT
cantrbry_lz4_32768_1_T.sqfs: GESCHEITERT
cantrbry_lz4_4096_1.sqfs: GESCHEITERT    
cantrbry_lz4_4096_1_T.sqfs: GESCHEITERT    
cantrbry_lz4_524288_1.sqfs: GESCHEITERT 
cantrbry_lz4_524288_1_T.sqfs: GESCHEITERT 
cantrbry_lz4_65536_1.sqfs: GESCHEITERT 
cantrbry_lz4_65536_1_T.sqfs: GESCHEITERT 
cantrbry_lz4_8192_1.sqfs: GESCHEITERT   
cantrbry_lz4_8192_1_T.sqfs: GESCHEITERT   
cantrbry_lzma_1048576_1.sqfs: GESCHEITERT
cantrbry_lzma_1048576_1_T.sqfs: GESCHEITERT
cantrbry_lzma_131072_1.sqfs: GESCHEITERT
cantrbry_lzma_131072_1_T.sqfs: GESCHEITERT
cantrbry_lzma_16384_1.sqfs: GESCHEITERT 
cantrbry_lzma_16384_1_T.sqfs: GESCHEITERT 
cantrbry_lzma_262144_1.sqfs: GESCHEITERT
cantrbry_lzma_262144_1_T.sqfs: GESCHEITERT
cantrbry_lzma_32768_1.sqfs: GESCHEITERT
cantrbry_lzma_32768_1_T.sqfs: GESCHEITERT
cantrbry_lzma_4096_1.sqfs: GESCHEITERT                     
cantrbry_lzma_4096_1_T.sqfs: GESCHEITERT                                                                               
cantrbry_lzma_524288_1.sqfs: GESCHEITERT
cantrbry_lzma_524288_1_T.sqfs: GESCHEITERT                 
cantrbry_lzma_65536_1.sqfs: GESCHEITERT
cantrbry_lzma_65536_1_T.sqfs: GESCHEITERT
cantrbry_lzma_8192_1.sqfs: GESCHEITERT                     
cantrbry_lzma_8192_1_T.sqfs: GESCHEITERT
cantrbry_lzo_1048576_1.sqfs: GESCHEITERT           
cantrbry_lzo_1048576_1_T.sqfs: GESCHEITERT         
cantrbry_lzo_131072_1.sqfs: GESCHEITERT
cantrbry_lzo_131072_1_T.sqfs: GESCHEITERT
cantrbry_lzo_16384_1.sqfs: GESCHEITERT
cantrbry_lzo_16384_1_T.sqfs: GESCHEITERT
cantrbry_lzo_262144_1.sqfs: GESCHEITERT
cantrbry_lzo_262144_1_T.sqfs: GESCHEITERT
cantrbry_lzo_32768_1.sqfs: GESCHEITERT
cantrbry_lzo_32768_1_T.sqfs: GESCHEITERT
cantrbry_lzo_4096_1.sqfs: GESCHEITERT
cantrbry_lzo_4096_1_T.sqfs: GESCHEITERT
cantrbry_lzo_524288_1.sqfs: GESCHEITERT
cantrbry_lzo_524288_1_T.sqfs: GESCHEITERT
cantrbry_lzo_65536_1.sqfs: GESCHEITERT
cantrbry_lzo_65536_1_T.sqfs: GESCHEITERT
cantrbry_lzo_8192_1.sqfs: GESCHEITERT
cantrbry_lzo_8192_1_T.sqfs: GESCHEITERT
cantrbry_xz_1048576_1.sqfs: GESCHEITERT
cantrbry_xz_1048576_1_T.sqfs: GESCHEITERT
cantrbry_xz_131072_1.sqfs: GESCHEITERT
cantrbry_xz_131072_1_T.sqfs: GESCHEITERT
cantrbry_xz_16384_1.sqfs: GESCHEITERT
cantrbry_xz_16384_1_T.sqfs: GESCHEITERT
cantrbry_xz_262144_1.sqfs: GESCHEITERT
cantrbry_xz_262144_1_T.sqfs: GESCHEITERT
cantrbry_xz_32768_1.sqfs: GESCHEITERT
cantrbry_xz_32768_1_T.sqfs: GESCHEITERT
cantrbry_xz_4096_1.sqfs: GESCHEITERT
cantrbry_xz_4096_1_T.sqfs: GESCHEITERT
cantrbry_xz_524288_1.sqfs: GESCHEITERT
cantrbry_xz_524288_1_T.sqfs: GESCHEITERT
cantrbry_xz_65536_1.sqfs: GESCHEITERT
cantrbry_xz_65536_1_T.sqfs: GESCHEITERT
cantrbry_xz_8192_1.sqfs: GESCHEITERT
cantrbry_xz_8192_1_T.sqfs: GESCHEITERT
cantrbry_zstd_1048576_1.sqfs: GESCHEITERT
cantrbry_zstd_1048576_1_T.sqfs: GESCHEITERT
cantrbry_zstd_131072_1.sqfs: GESCHEITERT
cantrbry_zstd_131072_1_T.sqfs: GESCHEITERT
cantrbry_zstd_16384_1.sqfs: GESCHEITERT
cantrbry_zstd_16384_1_T.sqfs: GESCHEITERT
cantrbry_zstd_262144_1.sqfs: GESCHEITERT
cantrbry_zstd_262144_1_T.sqfs: GESCHEITERT
cantrbry_zstd_32768_1.sqfs: GESCHEITERT
cantrbry_zstd_32768_1_T.sqfs: GESCHEITERT
cantrbry_zstd_4096_1.sqfs: GESCHEITERT
cantrbry_zstd_4096_1_T.sqfs: GESCHEITERT
cantrbry_zstd_524288_1.sqfs: GESCHEITERT
cantrbry_zstd_524288_1_T.sqfs: GESCHEITERT
cantrbry_zstd_65536_1.sqfs: GESCHEITERT
cantrbry_zstd_65536_1_T.sqfs: GESCHEITERT
cantrbry_zstd_8192_1.sqfs: GESCHEITERT
cantrbry_zstd_8192_1_T.sqfs: GESCHEITERT
sha512sum: WARNUNG: 108 berechnete Prüfsummen passten NICHT
sha512sum: /home/bdrung/projects/mmdebstrap/squashfs-tools-ng/tests/corpus/cantrbry.sha512: keine Datei wurde überprüft
FAIL tests/cantrbry.sh (exit status: 1)

FAIL: tests/test_tar_sqfs.sh
============================

@PaxHeader: unknown entry type
tests/tar/user-group-largenum/gnu.sqfs: GESCHEITERT
tests/tar/user-group-largenum/pax.sqfs: GESCHEITERT
tests/tar/user-group-largenum/8-digit.sqfs: GESCHEITERT
tests/tar/negative-mtime/gnu.sqfs: GESCHEITERT
tests/tar/negative-mtime/pax.sqfs: GESCHEITERT
tests/tar/large-mtime/gnu.sqfs: GESCHEITERT
tests/tar/large-mtime/pax.sqfs: GESCHEITERT
tests/tar/large-mtime/12-digit.sqfs: GESCHEITERT
tests/tar/xattr/xattr-schily-binary.sqfs: GESCHEITERT
tests/tar/xattr/xattr-schily.sqfs: GESCHEITERT
tests/tar/xattr/acl.sqfs: GESCHEITERT
tests/tar/xattr/fflags-schily.sqfs: GESCHEITERT
tests/tar/xattr/fflags-libarchive.sqfs: GESCHEITERT
tests/tar/xattr/xattr-libarchive.sqfs: GESCHEITERT
tests/tar/sparse-files/gnu.sqfs: GESCHEITERT
tests/tar/sparse-files/pax-gnu0-1.sqfs: GESCHEITERT
tests/tar/sparse-files/pax-gnu0-0.sqfs: GESCHEITERT
tests/tar/sparse-files/pax-gnu1-0.sqfs: GESCHEITERT
tests/tar/sparse-files/gnu-small.sqfs: GESCHEITERT
tests/tar/format-acceptance/gnu.sqfs: GESCHEITERT
tests/tar/format-acceptance/ustar.sqfs: GESCHEITERT
tests/tar/format-acceptance/pax.sqfs: GESCHEITERT
tests/tar/format-acceptance/gnu-g.sqfs: GESCHEITERT
tests/tar/format-acceptance/ustar-pre-posix.sqfs: GESCHEITERT
tests/tar/format-acceptance/v7.sqfs: GESCHEITERT
tests/tar/long-paths/gnu.sqfs: GESCHEITERT
tests/tar/long-paths/ustar.sqfs: GESCHEITERT
tests/tar/long-paths/pax.sqfs: GESCHEITERT
sha512sum: WARNUNG: 28 berechnete Prüfsummen passten NICHT
FAIL tests/test_tar_sqfs.sh (exit status: 1)

PS: I am running Ubuntu 19.10 with German locale as you can see in the output.

AgentD commented 4 years ago

I can cause all tests to fail on my system by setting SOURCE_DATE_EPOCH. Once again, it seems I overlooked that while the files have the time stamps from the inputs, the root inode is created with defaults (that can be altered by setting SOURCE_DATE_EPOCH).

Explicit --defaults mtime=0 in commit cae8192 fixes this on my end, but the problem with the differing LZ4/zstd versions remains.

bdrung commented 4 years ago

I pulled commit cae8192 and now all test cases succeed on Ubuntu 19.10.