♻️Storage: refactoring

[sanderegg]

What do these changes do?

very noisy refactoring before fixing storage DB connection locking issues. Basically moves the S3 client code into the aws-library and refactor related tests.

major changes:

• s3 client is now in the aws-library (see packages/aws-library/src/aws_library/s3/...)
• refactored s3 error handling decorators
• renamed all s3 client method file occurences to object
• added copy_objects_recursively method to copy folders in S3
• moved tests of s3 client in aws-library
• added fixture to create a folder with multiple files (see file_extra.py in pytest-simcore)
• old benchmark tests are removed (were never running anyway), but might reappear in a later PR

Related issue/s

How to test

Dev-ops checklist

• [x] No ENV changes or I properly updated ENV (read the instruction)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 93.17585% with 26 lines in your changes missing coverage. Please review.

Project coverage is 87.8%. Comparing base (cafbf96) to head (d1c1b1e). Report is 303 commits behind head on master.

Additional details and impacted files

[](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #5981 +/- ## ========================================= + Coverage 84.5% 87.8% +3.3% ========================================= Files 10 1419 +1409 Lines 214 58216 +58002 Branches 25 1395 +1370 ========================================= + Hits 181 51167 +50986 - Misses 23 6756 +6733 - Partials 10 293 +283 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.7% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.9% <93.1%> (+1.3%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [...ackages/aws-library/src/aws\_library/s3/\_\_init\_\_.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19faW5pdF9fLnB5) | `100.0% <100.0%> (ø)` | | | [...kages/aws-library/src/aws\_library/s3/\_constants.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_constants.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19jb25zdGFudHMucHk=) | `100.0% <100.0%> (ø)` | | | [packages/aws-library/src/aws\_library/s3/\_errors.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_errors.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19lcnJvcnMucHk=) | `100.0% <100.0%> (ø)` | | | [packages/aws-library/src/aws\_library/s3/\_models.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_models.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19tb2RlbHMucHk=) | `100.0% <100.0%> (ø)` | | | [packages/aws-library/src/aws\_library/s3/\_utils.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_utils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL191dGlscy5weQ==) | `100.0% <100.0%> (ø)` | | | [...-library/src/models\_library/api\_schemas\_storage.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=packages%2Fmodels-library%2Fsrc%2Fmodels_library%2Fapi_schemas_storage.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvbW9kZWxzLWxpYnJhcnkvc3JjL21vZGVsc19saWJyYXJ5L2FwaV9zY2hlbWFzX3N0b3JhZ2UucHk=) | `89.5% <100.0%> (ø)` | | | [...mcore\_service\_resource\_usage\_tracker/modules/s3.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=services%2Fresource-usage-tracker%2Fsrc%2Fsimcore_service_resource_usage_tracker%2Fmodules%2Fs3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvcmVzb3VyY2UtdXNhZ2UtdHJhY2tlci9zcmMvc2ltY29yZV9zZXJ2aWNlX3Jlc291cmNlX3VzYWdlX3RyYWNrZXIvbW9kdWxlcy9zMy5weQ==) | `90.0% <100.0%> (ø)` | | | [...\_tracker/services/resource\_tracker\_service\_runs.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=services%2Fresource-usage-tracker%2Fsrc%2Fsimcore_service_resource_usage_tracker%2Fservices%2Fresource_tracker_service_runs.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvcmVzb3VyY2UtdXNhZ2UtdHJhY2tlci9zcmMvc2ltY29yZV9zZXJ2aWNlX3Jlc291cmNlX3VzYWdlX3RyYWNrZXIvc2VydmljZXMvcmVzb3VyY2VfdHJhY2tlcl9zZXJ2aWNlX3J1bnMucHk=) | `100.0% <100.0%> (ø)` | | | [...storage/src/simcore\_service\_storage/application.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fapplication.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvYXBwbGljYXRpb24ucHk=) | `83.6% <100.0%> (ø)` | | | [...s/storage/src/simcore\_service\_storage/constants.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fconstants.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvY29uc3RhbnRzLnB5) | `100.0% <100.0%> (ø)` | | | ... and [10 more](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | | ... and [1371 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5981/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)