The current default strategy is to copy the entire raw dataset, file by file. The result is a potentially large number of AWS GET/POST requests that may be costly if the raw dataset contains many files. Change the default behavior to zip the raw dataset before sending to reduce size and the number of requests.
The current default strategy is to copy the entire raw dataset, file by file. The result is a potentially large number of AWS GET/POST requests that may be costly if the raw dataset contains many files. Change the default behavior to zip the raw dataset before sending to reduce size and the number of requests.