As I understand it, jobs are currently started without concurrency or delay settings, and those are later set by the settings monitor. This means that a job always starts at 1 concurrency and 0 delay. This may be a problem when targets have heavy rate limiting and even just two or three requests in a short time trigger that.
While setting the concurrency via the command line is broken (https://github.com/ArchiveTeam/wpull/issues/339), delays are fine (--wait), so the pipeline should use that. However, the command line options do not allow for specifying a custom range of delays, so the average of delay_min and delay_max should be used there.
As I understand it, jobs are currently started without concurrency or delay settings, and those are later set by the settings monitor. This means that a job always starts at 1 concurrency and 0 delay. This may be a problem when targets have heavy rate limiting and even just two or three requests in a short time trigger that.
While setting the concurrency via the command line is broken (https://github.com/ArchiveTeam/wpull/issues/339), delays are fine (
--wait), so the pipeline should use that. However, the command line options do not allow for specifying a custom range of delays, so the average ofdelay_minanddelay_maxshould be used there.