search

awslabs / soci-snapshotter

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.
Apache License 2.0
490 stars 54 forks source link

[FEATURE] Replace zlib with zlib-ng #521

Open Kern-- opened 1 year ago

Kern-- commented 1 year ago

Description

https://github.com/zlib-ng/zlib-ng

The soci snapshotter uses zlib for index generation / decompression. There is also zlib-ng, which contains optimizations for "next-generation" systems (such as SIMD, Intel and Cloudfare optimizations, et cetera).

From zlib-ng's README:

The motivation for this fork came after seeing several 3rd party contributions containing new optimizations not getting implemented into the official zlib repository.

Mark Adler has been maintaining zlib for a very long time, and he has done a great job and hopefully he will continue for a long time yet. The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change.

Early benchmarks using the time command and zran.c with various output sizes (100bytes - 10MiB) shows that zlib-ng performs 25%-60% better than zlib.

Describe the solution you'd like

1) SOCI uses zlib-ng instead of zlib for index generation and decompression at runtime 1) Run some amount of verification that the zlib-ng version of SOCI is compatible with non zlib version 1) Perform benchmarking to show the differences 1) Update docs on building against zlib-ng

Describe any alternative solutions/features you've considered

We have an alternative to rewrite our indexer in pure go: https://github.com/awslabs/soci-snapshotter/issues/76

Any additional context or information about the feature request

No response

Kern-- commented 1 year ago

For now, we want to create the proof of concept and benchmark it against the current zlib-based implementation.