Right now code and files are sent up and down between the computer running the script and each instance. This can create a lot of traffic when the volume of instances increases and doesn't scale well because upload speeds tend to be slow from people's PCs.
A better solution would be to use an S3 bucket. Files could be uploaded there once and then the instances told to read from that location.
In the same way results could be written to this location and the client given a url to the file but this would mean it would not be possible to concatenate and sort the files.
Right now code and files are sent up and down between the computer running the script and each instance. This can create a lot of traffic when the volume of instances increases and doesn't scale well because upload speeds tend to be slow from people's PCs.
A better solution would be to use an S3 bucket. Files could be uploaded there once and then the instances told to read from that location.
In the same way results could be written to this location and the client given a url to the file but this would mean it would not be possible to concatenate and sort the files.