Find fastest BZ2 decompress

JackWagner / stattrak

Counter-Strike 2 gameplay aggregator and stat collector

0 stars 0 forks source link

Find fastest BZ2 decompress #20

Closed JackWagner closed 10 months ago

JackWagner commented 10 months ago

BACKGROUND

Context: BZ2 decompression is the current bottleneck in the demo ETL
Problem: We need a faster way to unzip/decompress a BZ2 file
Solution: Test speed of different BZ2 decompression approaches

STEPS:

[ ] ___
[ ] ___
[ ] ___

JackWagner commented 10 months ago

lbzip2 seems like the most viable option, it is a parallel decompressor. However, it is apart of the Extra Packages for Enterprise Linux (or EPEL) repo for Linux which is not supported by Amazon Linux 3 OS but is by Amazon Linux 2 (super lame).

Will have to modify the EC2 instance to be AL2 and reconfigure everything