The downloading of web pages has a variable rate limit, and new workers are created to reach to desired limit. Downloading lists of CDX records (snapshot pages) can take several seconds, and with a high-enough webpage download rate limit, it could end up making the snapshot page downloading the limiting factor.
We should try to rewrite the snapshot downloading so that it can also be parallelised.
The downloading of web pages has a variable rate limit, and new workers are created to reach to desired limit. Downloading lists of CDX records (snapshot pages) can take several seconds, and with a high-enough webpage download rate limit, it could end up making the snapshot page downloading the limiting factor.
We should try to rewrite the snapshot downloading so that it can also be parallelised.