eranrund
commented
1 year ago The apps you mentioned above are all using the ledger sync crate, which takes advantage of these roll up blocks (called merged blocks in the codebase). See https://github.com/mobilecoinfoundation/mobilecoin/blob/master/ledger/sync/src/reqwest_transactions_fetcher.rs#L254
We're not using offical S3 SDKs and while it looks like the crate we're using supports multipart uploads, we're not taking advantage of that. I'm not sure how much that matters the large merged blocks don't appear too frequently right now at our current average block rate. For downloading we are also not doing multipart downloads and we might see better performance if we did, since ledger syncs that start from an empty ledger or one that is far enough behind do take advantage of the large merged files. It's possible that we'd benefit from bigger files (e.g. 100k/1M).
We have single, 1000 and 10000 rollsup blocks, is this the most efficient? Are our apps that download ledger using these options?
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html#optimizing-performance-guidelines-get-range
Will the sdk do some optimization for us if we have larger files? Is it smart enough to do multi-part uploads/downloads and/or byte ranges?
Should we do 100k and 1M block roll ups?