search

awslabs / aws-c-s3

C99 library implementation for communicating with the S3 service, designed for maximizing throughput on high bandwidth EC2 instances.
Apache License 2.0
94 stars 38 forks source link

Parallel prepare - open file on needed #355

Closed TingDaoK closed 11 months ago

TingDaoK commented 11 months ago

Parallel preparation, the parallel stream will open the file when reading from it and seek to the right offset to read, and close the file after reading.

We checked it has no performance impact with mmap impl, and it's simpler. Simpler is better

With this approach, the user of the parallel input stream will have to manage the threads and concurrence, but it's only used by our s3 client, which already handles the concurrence and thread pool, we are good.

Original: https://github.com/awslabs/aws-c-s3/pull/353

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter commented 11 months ago

Codecov Report

Merging #355 (68aa652) into main (76c6d9f) will decrease coverage by 0.01%. The diff coverage is 93.54%.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355/graphs/tree.svg?width=650&height=150&src=pr&token=J4KP54FVLF&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs)](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #355 +/- ## ========================================== - Coverage 89.59% 89.58% -0.01% ========================================== Files 17 18 +1 Lines 4929 4982 +53 ========================================== + Hits 4416 4463 +47 - Misses 513 519 +6 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzLmM=) | `96.15% <ø> (ø)` | | | [source/s3\_auto\_ranged\_put.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2F1dG9fcmFuZ2VkX3B1dC5j) | `92.10% <100.00%> (-0.05%)` | :arrow_down: | | [source/s3\_default\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2RlZmF1bHRfbWV0YV9yZXF1ZXN0LmM=) | `94.63% <100.00%> (+0.03%)` | :arrow_up: | | [source/s3\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX21ldGFfcmVxdWVzdC5j) | `92.83% <100.00%> (+0.07%)` | :arrow_up: | | [source/s3\_request\_messages.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX3JlcXVlc3RfbWVzc2FnZXMuYw==) | `73.75% <ø> (-1.02%)` | :arrow_down: | | [source/s3\_parallel\_input\_stream.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX3BhcmFsbGVsX2lucHV0X3N0cmVhbS5j) | `89.28% <89.28%> (ø)` | |