klauspost / compress

Optimized Go Compression Packages
Other
4.68k stars 311 forks source link

Decompressing gzip: `invalid header` #972

Closed rgmz closed 3 months ago

rgmz commented 3 months ago

I've encountered what I believe is a valid GZIP file. However, attempting to read the contents results in the error gzip: invalid header. It's not clear to me what the error is, as the file is successfully decompressed with both gunzip and Java (see examples below).

Steps to Reproduce

  1. Download https://github.com/akuang/self-driving-car-sim/blob/4b1f739ebda9ed4920fe895ee3677bd4ccb79218/Assets/Standard%20Assets/Environment/SpeedTree/Conifer/Conifer_Desktop.spm
  2. Run the following reproducer code.

    import (
        "io"
        "os"
        "testing"
    
        "github.com/klauspost/compress/gzip"
        "github.com/stretchr/testify/require"
    )
    
    func TestGzip(t *testing.T) {
        f, err := os.Open("/tmp/Conifer_Desktop.spm")
        require.NoError(t, err)
        defer f.Close()
    
        rc, err := gzip.NewReader(f)
        require.NoError(t, err)
        defer rc.Close()
    
        _, err = io.ReadAll(rc)
        require.NoError(t, err)
    }
  3. Observe the error invalid header
    $ go test gzip_test.go 
    --- FAIL: TestGzip (0.01s)
        main_test.go:22:
                    Error Trace:    /tmp/repro/gzip_test.go:22
                    Error:          Received unexpected error:
                                    gzip: invalid header
                    Test:           TestGzip
    FAIL
    FAIL    command-line-arguments  0.017s
    FAIL

Working Examples

gunzip

There is a warning about trailing garbage which may be relevant.

$ file -i Conifer_Desktop.spm
Conifer_Desktop.spm: application/gzip; charset=binary
$ gunzip -f Conifer_Desktop.spm -S .spm

gzip: Conifer_Desktop.spm: decompression OK, trailing garbage ignored

Java Code

import java.io.*;
import java.util.zip.GZIPInputStream;

public class Main {

    public static void main(String[] args) {
        var in = "C:\\Users\\Me\\Downloads\\Conifer_Desktop.spm";
        var out = STR."\{in}.out";

        decompressGzip(in, out);
    }

    public static void decompressGzip(String inputFile, String outputFile) {

        try (var gis = new GZIPInputStream(new FileInputStream(inputFile));
             var fos = new FileOutputStream(outputFile)) {

            // copy GZIPInputStream to FileOutputStream
            byte[] buffer = new byte[1024];
            int len;
            while ((len = gis.read(buffer)) > 0) {
                fos.write(buffer, 0, len);
            }
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }
}
klauspost commented 3 months ago

See https://pkg.go.dev/github.com/klauspost/compress/gzip#Reader.Multistream

Add rc.Multistream(false)