fix(attachment): convoy separation

Problem

Closes FRM-1757

Admins are not able to receive all the download responses when clicking download with CSV.

Most modern browser blocks sequential downloads that run in quick successions.

Symptoms

results are mainly without attachments
responses > 10

Solution

Implementation of convoy separation.

Downloads are grouped into convoys where the front of the convoy must be x ms away from the next convoy.

[ o . o . . o o o ] . . [ o o o . . o . o] [ o o o o ]
  ^                       ^                  ^
  1st convoy              2nd convoy         3rd convoy                  

CONVOY_SIZE = 5
The time between each convoy should X amount of distance away to avoid a flood
3rd Convoy is too close and thus will be forced to wait

[ o . o . . o o o ] . . [ o o o . . o . o] . . . . [ o o o o ]
  ^                       ^                          ^
  1st convoy              2nd convoy                 3rd convoy

Additional Context

Total responses = 44 Experiments:

Control Group

no delay -> get 20 zips

Batch delay

1000ms of delay every 7 files -> get all 44 zips (increased completion time of 6 seconds)
200ms of delay every 7 files -> get 22 zips

Convoy delay (CONVOY_SIZE = 7)

1000ms between first of next convoy, -> get 39 zips

Convoy delay (CONVOY_SIZE = 5)

1000ms between first of next convoy, -> get all 44 zips
Breaking Changes
No - this PR is backwards compatible

Tests

Regression on large submissions >10 crossing two convoys

[ ] Create 40 submissions with 20 attachments
[ ] Ensure that Download CSV (without attachment) returns 1 .csv file
[ ] Ensure that Download CSV with attachment returns 40 .zip and 1 .csv file

Regression on small submissions < 10

[ ] Create 7 submissions with 3 attachments only 1 convoy
[ ] Ensure that Download CSV (without attachment) returns 1 .csv file
[ ] Ensure that Download CSV with attachment returns 7 .zip and 1 .csv file

opengovsg / FormSG