Open newsch opened 1 year ago
With more optimization it's likely that decompressing the gzip archives will be a bottleneck. There are a number of "parallel" implementations for gzip, but all the ones I've looked at aren't actually useful for this case.
Steps:
What I've looked at so far:
Let's run it first in the current state and see real base numbers before prioritizing/digging deeper into any optimizations.
With more optimization it's likely that decompressing the gzip archives will be a bottleneck. There are a number of "parallel" implementations for gzip, but all the ones I've looked at aren't actually useful for this case.
Steps:
What I've looked at so far: