Closed sillitoe closed 1 year ago
introduce a loop that unzips/processes each archive file separately. This should spread the network io over random periods of the whole job, rather than all of it happening at the start. Should also allow us to checkpoint each archive file.
introduce a loop that unzips/processes each archive file separately. This should spread the network io over random periods of the whole job, rather than all of it happening at the start. Should also allow us to checkpoint each archive file.