search

hhvm / packaging

The sources for how we have built the HHVM packages.
MIT License
65 stars 65 forks source link

`aws s3 sync` is slow #312

Open Atry opened 2 years ago

Atry commented 2 years ago

We use aws s3 sync to upload deb packages from EBS to S3 https://github.com/hhvm/packaging/blob/6d301967b15491b4c504947ec9bf0215ffb1a803/bin/update-debianish-repo#L49-L50

According to the document, aws s3 sync will compare the content of all the 2TB files even when we just added 5GB files. It would be good to avoid checking the 2TB files.

Atry commented 2 years ago

Even though it is still not ideal, I think Fred had partially optimized it, because previously we had to downloading the 2TB files before the updating, which is worse.

Atry commented 2 years ago

Why not mount S3 with s3fs and directly run reprepro in the mounted path, instead of aws s3 sync?