seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

perf: use libdeflate to improve gzip decompression #28

Closed william-silversmith closed 3 years ago

william-silversmith commented 3 years ago

This significantly improves certain workloads such as unsharded segmentation.

https://github.com/dcwatson/deflate/pull/3