Adds Connection Pool Destroy Delay Take 2

[waahm7]

Description of changes: Problem: If the user makes requests serially, we destroy the connection pool between each request, which impacts performance.

Solution: Periodically run a cleanup task which cleans up the unused S3 endpoints instead of triggering a sync cleanup.

Benchmark: Downloading 1000 64Kb files in serial on c5n.18xlarge in crt-c.

Main Branch:

Run:1 Secs:42.352150 Gb/s:0.012379
Run:2 Secs:39.855370 Gb/s:0.013155
Run:3 Secs:40.601442 Gb/s:0.012913
Run:4 Secs:45.805565 Gb/s:0.011446
Run:5 Secs:40.032408 Gb/s:0.013097
Run:6 Secs:40.855503 Gb/s:0.012833
Run:7 Secs:45.984822 Gb/s:0.011401
Run:8 Secs:42.299984 Gb/s:0.012395
Run:9 Secs:45.083880 Gb/s:0.011629
Run:10 Secs:40.494285 Gb/s:0.012947
Overall Throughput (Gb/s) Median:0.012614 Mean:0.012419 Min:0.011401 Max:0.013155 Variance:0.000000 StdDev:0.000656
Overall Duration (Secs) Median:41.577743 Mean:42.336541 Min:39.855370 Max:45.984822 Variance:5.295672 StdDev:2.301233
Peak RSS:20.636719 MiB

This branch:
Run:1 Secs:23.644483 Gb/s:0.022174
Run:2 Secs:19.403419 Gb/s:0.027020
Run:3 Secs:18.151826 Gb/s:0.028883
Run:4 Secs:22.688095 Gb/s:0.023109
Run:5 Secs:18.038438 Gb/s:0.029065
Run:6 Secs:19.108228 Gb/s:0.027438
Run:7 Secs:17.936860 Gb/s:0.029230
Run:8 Secs:17.901290 Gb/s:0.029288
Run:9 Secs:18.746993 Gb/s:0.027967
Run:10 Secs:17.442144 Gb/s:0.030059
Overall Throughput (Gb/s) Median:0.028425 Mean:0.027423 Min:0.022174 Max:0.030059 Variance:0.000007 StdDev:0.002552
Overall Duration (Secs) Median:18.449410 Mean:19.306178 Min:17.442144 Max:23.644483 Variance:4.088264 StdDev:2.021946

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.49%. Comparing base (0ce756e) to head (a25f9ad).

:exclamation: Current head a25f9ad differs from pull request most recent head 7aa108d. Consider uploading reports for the commit 7aa108d to get more accurate results

Additional details and impacted files

[](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/419?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #419 +/- ## ========================================== + Coverage 89.40% 89.49% +0.08% ========================================== Files 20 20 Lines 5872 5911 +39 ========================================== + Hits 5250 5290 +40 + Misses 622 621 -1 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/419?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3\_client.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/419?src=pr&el=tree&filepath=source%2Fs3_client.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2NsaWVudC5j) | `88.61% <100.00%> (+0.45%)` | :arrow_up: | | [source/s3\_endpoint.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/419?src=pr&el=tree&filepath=source%2Fs3_endpoint.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2VuZHBvaW50LmM=) | `90.97% <100.00%> (+0.60%)` | :arrow_up: |