Good day! I have a huge file collection(about 3 million files) and have uploaded around half of it but I stop midway because of some issue(s) in the EC2 server.
Now, I tried to restart the process and I understand that it will again compare and skip out all the unmodified file(s) but it seems like it is taking a lot of time on just listing and comparing the files and I am curious if this is normal? I am currently at 400000 files as listed in the logs and this has been running close to two days already.
I have the following command executed just for context:
"nohup bash -c './mtglacier sync --config config.d/sync-media-backup.cfg --max-number-of-files 3000000 --concurrency 4 --partsize 64 --new >> /var/log/aws-glacier/sync.log' > /dev/null 2>&1 & echo $!"
Additionally, the EC2 server I have is also newly created and is dedicated just for this glacier upload operation. Is their a configuration option that I am missing to somehow throttle the sync process? Appreciate any feedback!
Good day! I have a huge file collection(about 3 million files) and have uploaded around half of it but I stop midway because of some issue(s) in the EC2 server. Now, I tried to restart the process and I understand that it will again compare and skip out all the unmodified file(s) but it seems like it is taking a lot of time on just listing and comparing the files and I am curious if this is normal? I am currently at 400000 files as listed in the logs and this has been running close to two days already.
I have the following command executed just for context: "nohup bash -c './mtglacier sync --config config.d/sync-media-backup.cfg --max-number-of-files 3000000 --concurrency 4 --partsize 64 --new >> /var/log/aws-glacier/sync.log' > /dev/null 2>&1 & echo $!"
Additionally, the EC2 server I have is also newly created and is dedicated just for this glacier upload operation. Is their a configuration option that I am missing to somehow throttle the sync process? Appreciate any feedback!