zip-rs / zip-old

Zip implementation in Rust
MIT License
731 stars 204 forks source link

prototype async API, with demonstrable perf improvements via benchmark #409

Closed cosmicexplorer closed 5 months ago

cosmicexplorer commented 1 year ago

This is another response to zip-rs/zip2#165.

After demonstrating an approach to parallelize extracting zip archives using rayon threadpools in zip-rs/zip-old#407, I wanted to further investigate whether introducing an async API might provide a more scalable approach to introducing parallelism (as I noted in https://github.com/zip-rs/zip/issues/403#issuecomment-1728747153). This PR prototypes such an API.

Example Usage

See the doctest: https://github.com/zip-rs/zip/blob/bc601d744bf8f61cb3eda88d39610de55098b120/src/read/tokio.rs#L554-L576

Performance & Benchmarking

cargo bench -- io demonstrates that this async implementation is faster to extract for large archives! ```bash > cargo bench -- io io/big archive(84642745 bytes)/ time: [29.463 ms 29.661 ms 29.865 ms] thrpt: [2.6396 GiB/s 2.6577 GiB/s 2.6756 GiB/s] io/big archive(84642745 bytes)/ time: [72.790 ms 73.435 ms 74.127 ms] thrpt: [1.0634 GiB/s 1.0735 GiB/s 1.0830 GiB/s] io/big archive(84642745 bytes)/ time: [35.737 ms 35.870 ms 36.016 ms] thrpt: [2.1887 GiB/s 2.1976 GiB/s 2.2058 GiB/s] io/small archive(126517 bytes)/ time: [113.44 µs 114.66 µs 115.86 µs] thrpt: [1.0170 GiB/s 1.0276 GiB/s 1.0386 GiB/s] io/small archive(126517 bytes)/ time: [167.58 µs 169.65 µs 171.53 µs] thrpt: [703.41 MiB/s 711.20 MiB/s 720.00 MiB/s] io/small archive(126517 bytes)/ time: [60.929 µs 61.357 µs 61.835 µs] thrpt: [1.9055 GiB/s 1.9204 GiB/s 1.9339 GiB/s] io/random archive(10106802 bytes)/ time: [4.2838 ms 4.3385 ms 4.4067 ms] thrpt: [2.1360 GiB/s 2.1696 GiB/s 2.1973 GiB/s] io/random archive(10106802 bytes)/ time: [9.4273 ms 9.5448 ms 9.6646 ms] thrpt: [997.31 MiB/s 1009.8 MiB/s 1022.4 MiB/s] io/random archive(10106802 bytes)/ time: [3.9525 ms 3.9812 ms 4.0152 ms] thrpt: [2.3443 GiB/s 2.3643 GiB/s 2.3814 GiB/s] extract/big archive(84642745 bytes)/ time: [779.27 ms 801.90 ms 824.78 ms] thrpt: [97.870 MiB/s 100.66 MiB/s 103.59 MiB/s] extract/big archive(84642745 bytes)/ time: [802.74 ms 803.94 ms 805.23 ms] thrpt: [100.25 MiB/s 100.41 MiB/s 100.56 MiB/s] extract/small archive(126517 bytes)/ time: [9.5155 ms 9.5789 ms 9.6337 ms] thrpt: [12.524 MiB/s 12.596 MiB/s 12.680 MiB/s] extract/small archive(126517 bytes)/ time: [2.3227 ms 2.3490 ms 2.3713 ms] thrpt: [50.881 MiB/s 51.365 MiB/s 51.946 MiB/s] extract/random archive(10106802 bytes)/ time: [160.18 ms 164.26 ms 169.09 ms] thrpt: [57.004 MiB/s 58.681 MiB/s 60.173 MiB/s] extract/random archive(10106802 bytes)/ time: [37.677 ms 37.850 ms 38.045 ms] thrpt: [253.35 MiB/s 254.65 MiB/s 255.82 MiB/s] ```

Notably, this approach is already faster than the sync implementation, without even applying the optimizations requiring a cloneable file handle that were necessary in zip-rs/zip-old#407!

TODO

This is left as a draft because it adds a lot of extra dependencies, which I would like to avoid. It also directly depends on tokio, instead of using the async-executors crate to apply to any async executor. I'm going to spend some time to break out the extra code I had to write to make this async API work into a separate wrapper crate, which should allow this change to become much smaller and easier for maintainers to accept.

Pr0methean commented 5 months ago

Replaced with https://github.com/zip-rs/zip2/pull/73.