🐛Storage: fix database locking

[sanderegg]

What do these changes do?

This PR aims at removing all unbounded database connection locks in storage

Details

• in s3 client copy_objects_recursively removed deletion in case of error as this could delete fils in an uncontrolled way
• typing improvements in aws-library
• better logging in storage

Related issue/s

• fixes https://github.com/ITISFoundation/osparc-simcore/issues/5976

  Dev-ops checklist

  • [x] No ENV changes or I properly updated ENV (read the instruction)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 89.80892% with 16 lines in your changes missing coverage. Please review.

Project coverage is 87.7%. Comparing base (cafbf96) to head (1a31807). Report is 310 commits behind head on master.

Additional details and impacted files

[](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #6005 +/- ## ========================================= + Coverage 84.5% 87.7% +3.2% ========================================= Files 10 1330 +1320 Lines 214 55249 +55035 Branches 25 1368 +1343 ========================================= + Hits 181 48501 +48320 - Misses 23 6465 +6442 - Partials 10 283 +273 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.8% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.7% <89.8%> (+1.1%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [packages/aws-library/src/aws\_library/s3/\_client.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_client.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19jbGllbnQucHk=) | `96.7% <100.0%> (ø)` | | | [...src/simcore\_service\_storage/handlers\_simcore\_s3.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fhandlers_simcore_s3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvaGFuZGxlcnNfc2ltY29yZV9zMy5weQ==) | `100.0% <100.0%> (ø)` | | | [...s/aws-library/src/aws\_library/s3/\_error\_handler.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fs3%2F_error_handler.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L3MzL19lcnJvcl9oYW5kbGVyLnB5) | `76.1% <60.0%> (ø)` | | | [packages/service-library/src/servicelib/utils.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&filepath=packages%2Fservice-library%2Fsrc%2Fservicelib%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvc2VydmljZS1saWJyYXJ5L3NyYy9zZXJ2aWNlbGliL3V0aWxzLnB5) | `91.6% <80.0%> (ø)` | | | [...rage/src/simcore\_service\_storage/simcore\_s3\_dsm.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fsimcore_s3_dsm.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2Uvc2ltY29yZV9zM19kc20ucHk=) | `89.8% <91.1%> (ø)` | | ... and [1312 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6005/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)