valyala / gozstd

go wrapper for zstd
MIT License
420 stars 60 forks source link

StreamCompress() returns different from Compress() result #27

Closed kaxap closed 4 years ago

kaxap commented 4 years ago

go version: 1.14.2

Test:

func TestStream(t *testing.T) {
    buffer := []byte("this is test data for StreamCompress vs Compress comparison")
    reader := bytes.NewReader(buffer)
    writer := bytes.NewBuffer(nil)
    err := zstd.StreamCompress(writer, reader)
    if err != nil {
        t.Fatal(err)
    }
    streamResult := writer.Bytes()
    compressResult := zstd.Compress(nil, buffer)
    assert.Equal(t, streamResult, compressResult)
}

fails with

=== RUN   TestStream
    assert.go:24: got [40 181 47 253 0 88 216 1 0 116 104 105 115 32 105 115 32 116 101 115 116 32 100 97 116 97 32 102 111 114 32 83 116 114 101 97 109 67 111 109 112 114 101 115 115 32 118 115 32 67 111 109 112 114 101 115 115 32 99 111 109 112 97 114 105 115 111 110 1 0 0] want [40 181 47 253 32 59 217 1 0 116 104 105 115 32 105 115 32 116 101 115 116 32 100 97 116 97 32 102 111 114 32 83 116 114 101 97 109 67 111 109 112 114 101 115 115 32 118 115 32 67 111 109 112 114 101 115 115 32 99 111 109 112 97 114 105 115 111 110] 

--- FAIL: TestStream (0.00s)
FAIL

Seems like StreamCompress adds 0x01, 0x00, 0x00 bytes at the end.

valyala commented 4 years ago

The compressed data may differ when using streamed compression vs block compression, since streamed compression may split the data into multiple frames. You need to make sure that both stream-compressed and block-compressed data uncompress into the same data.

kaxap commented 4 years ago

I see, thanks for the explanation.