wangkuiyi / recordio

Apache License 2.0
11 stars 2 forks source link

Reduce buffering a chunk before compress it #53

Closed wangkuiyi closed 5 years ago

wangkuiyi commented 5 years ago

Currently, when function chunk.dump writes a chunk into the file, it writes the raw representation into a buffer data before compressing it and writing into the file. However, data is not necessary; we can compress on-the-fly into the compressed buffer directly without the need for data.