For large file lists we're firing off like 40000 promises at once. Node's async I/O is supposed to handle this, but it seems like at these numbers everything falls apart. We open hundreds of file descriptors, and requests start to fail because we're absolutely hammering the backend and perhaps even overwhelming the sending machine. This change awaits each upload requests so there's at most 16 going simultaneously instead of an indeterminate amount.
For large file lists we're firing off like 40000 promises at once. Node's async I/O is supposed to handle this, but it seems like at these numbers everything falls apart. We open hundreds of file descriptors, and requests start to fail because we're absolutely hammering the backend and perhaps even overwhelming the sending machine. This change awaits each upload requests so there's at most 16 going simultaneously instead of an indeterminate amount.