pierrec / lz4

LZ4 compression and decompression in pure Go
BSD 3-Clause "New" or "Revised" License
878 stars 142 forks source link

Did not return byte buffer when reader read to EOF ( from v4.1.1) #221

Open aiquestion opened 2 months ago

aiquestion commented 2 months ago

We use reader to decompress with no pooling object, it's some code like:

func Decompress(compressed []byte) ([]byte, error) {
    if len(compressed) == 0 {
        return compressed, nil
    }
    destBuffer := bytes.Buffer{}
    bytesReader := bytes.NewReader(compressed)
    lz4Reader := lz4.NewReader(bytesReader)
    if _, err := destBuffer.ReadFrom(lz4Reader); err != nil {
        return nil, err
    }
    return destBuffer.Bytes(), nil
}

and we found a big performace regression when we upgrade to v4.1.21 (from v4.0.2), after some test we narrow down the change from v4.1.0 -> v4.1.1.

It's seems that the code do BlockSizeIndex.Get() twice and only call Put() to return the buffer 1 time. It look like that the buffer from this stack is not released.

lz4block.BlockSizeIndex.Get (blocks.go:54) github.com/pierrec/lz4/v4/internal/lz4block
lz4stream.NewFrameDataBlock (block.go:197) github.com/pierrec/lz4/v4/internal/lz4stream
lz4stream.(*Blocks).initR (block.go:99) github.com/pierrec/lz4/v4/internal/lz4stream
lz4stream.(*Frame).InitR (frame.go:114) github.com/pierrec/lz4/v4/internal/lz4stream
lz4.(*Reader).init (reader.go:91) github.com/pierrec/lz4/v4
lz4.(*Reader).Read (reader.go:111) github.com/pierrec/lz4/v4
bytes.(*Buffer).ReadFrom (buffer.go:211) bytes

Tried to fix it in this PR https://github.com/pierrec/lz4/pull/220, pls help to review