After demonstrating an approach to parallelize extracting zip archives using rayon threadpools in zip-rs/zip-old#407, I wanted to further investigate whether introducing an async API might provide a more scalable approach to introducing parallelism (as I noted in https://github.com/zip-rs/zip/issues/403#issuecomment-1728747153). This PR prototypes such an API.
cargo bench -- io demonstrates that this async implementation is faster to extract for large archives!
```bash
> cargo bench -- io
io/big archive(84642745 bytes)/
time: [29.463 ms 29.661 ms 29.865 ms]
thrpt: [2.6396 GiB/s 2.6577 GiB/s 2.6756 GiB/s]
io/big archive(84642745 bytes)/
time: [72.790 ms 73.435 ms 74.127 ms]
thrpt: [1.0634 GiB/s 1.0735 GiB/s 1.0830 GiB/s]
io/big archive(84642745 bytes)/
time: [35.737 ms 35.870 ms 36.016 ms]
thrpt: [2.1887 GiB/s 2.1976 GiB/s 2.2058 GiB/s]
io/small archive(126517 bytes)/
time: [113.44 µs 114.66 µs 115.86 µs]
thrpt: [1.0170 GiB/s 1.0276 GiB/s 1.0386 GiB/s]
io/small archive(126517 bytes)/
time: [167.58 µs 169.65 µs 171.53 µs]
thrpt: [703.41 MiB/s 711.20 MiB/s 720.00 MiB/s]
io/small archive(126517 bytes)/
time: [60.929 µs 61.357 µs 61.835 µs]
thrpt: [1.9055 GiB/s 1.9204 GiB/s 1.9339 GiB/s]
io/random archive(10106802 bytes)/
time: [4.2838 ms 4.3385 ms 4.4067 ms]
thrpt: [2.1360 GiB/s 2.1696 GiB/s 2.1973 GiB/s]
io/random archive(10106802 bytes)/
time: [9.4273 ms 9.5448 ms 9.6646 ms]
thrpt: [997.31 MiB/s 1009.8 MiB/s 1022.4 MiB/s]
io/random archive(10106802 bytes)/
time: [3.9525 ms 3.9812 ms 4.0152 ms]
thrpt: [2.3443 GiB/s 2.3643 GiB/s 2.3814 GiB/s]
extract/big archive(84642745 bytes)/
time: [779.27 ms 801.90 ms 824.78 ms]
thrpt: [97.870 MiB/s 100.66 MiB/s 103.59 MiB/s]
extract/big archive(84642745 bytes)/
time: [802.74 ms 803.94 ms 805.23 ms]
thrpt: [100.25 MiB/s 100.41 MiB/s 100.56 MiB/s]
extract/small archive(126517 bytes)/
time: [9.5155 ms 9.5789 ms 9.6337 ms]
thrpt: [12.524 MiB/s 12.596 MiB/s 12.680 MiB/s]
extract/small archive(126517 bytes)/
time: [2.3227 ms 2.3490 ms 2.3713 ms]
thrpt: [50.881 MiB/s 51.365 MiB/s 51.946 MiB/s]
extract/random archive(10106802 bytes)/
time: [160.18 ms 164.26 ms 169.09 ms]
thrpt: [57.004 MiB/s 58.681 MiB/s 60.173 MiB/s]
extract/random archive(10106802 bytes)/
time: [37.677 ms 37.850 ms 38.045 ms]
thrpt: [253.35 MiB/s 254.65 MiB/s 255.82 MiB/s]
```
Notably, this approach is already faster than the sync implementation, without even applying the optimizations requiring a cloneable file handle that were necessary in zip-rs/zip-old#407!
TODO
This is left as a draft because it adds a lot of extra dependencies, which I would like to avoid. It also directly depends on tokio, instead of using the async-executors crate to apply to any async executor. I'm going to spend some time to break out the extra code I had to write to make this async API work into a separate wrapper crate, which should allow this change to become much smaller and easier for maintainers to accept.
This is another response to zip-rs/zip2#165.
After demonstrating an approach to parallelize extracting zip archives using rayon threadpools in zip-rs/zip-old#407, I wanted to further investigate whether introducing an
async
API might provide a more scalable approach to introducing parallelism (as I noted in https://github.com/zip-rs/zip/issues/403#issuecomment-1728747153). This PR prototypes such an API.Example Usage
See the doctest: https://github.com/zip-rs/zip/blob/bc601d744bf8f61cb3eda88d39610de55098b120/src/read/tokio.rs#L554-L576
Performance & Benchmarking
cargo bench -- io demonstrates that this async implementation is faster to extract for large archives!
```bash > cargo bench -- io io/big archive(84642745 bytes)/Notably, this approach is already faster than the sync implementation, without even applying the optimizations requiring a cloneable file handle that were necessary in zip-rs/zip-old#407!
TODO
This is left as a draft because it adds a lot of extra dependencies, which I would like to avoid. It also directly depends on
tokio
, instead of using theasync-executors
crate to apply to any async executor. I'm going to spend some time to break out the extra code I had to write to make this async API work into a separate wrapper crate, which should allow this change to become much smaller and easier for maintainers to accept.