I am wondering what is the best practice to use knox to do that effectively. At the moment, I am simply writing the files one by one and observing that after the first 4000 PUTs the process becomes very slow. My guess is that there is some "buffer" of PUT requests that can't get processed fast enough.
Hi All, I need to create almost 2m small files in an S3 bucket (see the
make.js
script at https://github.com/Digital-Contraptions-Imaginarium/frugal-addresses-onspd/tree/write-to-s3-directly if you are curious).I am wondering what is the best practice to use knox to do that effectively. At the moment, I am simply writing the files one by one and observing that after the first 4000 PUTs the process becomes very slow. My guess is that there is some "buffer" of PUT requests that can't get processed fast enough.
Any advice is welcome, thanks,
Giacecco