🎨 Increase performance by paginating `search files` entry-points

pcrespov commented 2 months ago

What do these changes do?

This PR adds pagination options to storage's :search entry-point (storage api version: 0.3.0 → 0.4.0). Then it limits the search to a single file in the api-server handlers.

As a result of profiling storage we noticed that the file listings are not bound and therefore it slows down the queries as the number of result files increases creating a chain of failures starting with client timeouts.

Related issue/s

Improvement after running diagnostics for #5694
resolves https://github.com/ITISFoundation/osparc-simcore/issues/5738
resolves https://github.com/ITISFoundation/osparc-simcore/issues/5724

How to test

Driving tests services/storage/tests/unit/test_handlers_simcore_s3.py

Dev-ops checklist

[x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 68.7%. Comparing base (cafbf96) to head (f70a92e). Report is 191 commits behind head on master.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796/graphs/tree.svg?width=650&height=150&src=pr&token=h1rOE8q7ic&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #5796 +/- ## ========================================= - Coverage 84.5% 68.7% -15.9% ========================================= Files 10 666 +656 Lines 214 32741 +32527 Branches 25 205 +180 ========================================= + Hits 181 22501 +22320 - Misses 23 10188 +10165 - Partials 10 52 +42 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `65.2% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.0% <100.0%> (+5.4%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [...src/simcore\_service\_api\_server/api/routes/files.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fapi-server%2Fsrc%2Fsimcore_service_api_server%2Fapi%2Froutes%2Ffiles.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXBpLXNlcnZlci9zcmMvc2ltY29yZV9zZXJ2aWNlX2FwaV9zZXJ2ZXIvYXBpL3JvdXRlcy9maWxlcy5weQ==) | `78.3% <ø> (ø)` | | | [...vice\_api\_server/api/routes/solvers\_jobs\_getters.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fapi-server%2Fsrc%2Fsimcore_service_api_server%2Fapi%2Froutes%2Fsolvers_jobs_getters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXBpLXNlcnZlci9zcmMvc2ltY29yZV9zZXJ2aWNlX2FwaV9zZXJ2ZXIvYXBpL3JvdXRlcy9zb2x2ZXJzX2pvYnNfZ2V0dGVycy5weQ==) | `92.9% <ø> (ø)` | | | [...src/simcore\_service\_api\_server/services/storage.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fapi-server%2Fsrc%2Fsimcore_service_api_server%2Fservices%2Fstorage.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXBpLXNlcnZlci9zcmMvc2ltY29yZV9zZXJ2aWNlX2FwaV9zZXJ2ZXIvc2VydmljZXMvc3RvcmFnZS5weQ==) | `69.0% <100.0%> (ø)` | | | [...e/src/simcore\_service\_storage/db\_file\_meta\_data.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fdb_file_meta_data.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvZGJfZmlsZV9tZXRhX2RhdGEucHk=) | `98.6% <100.0%> (ø)` | | | [...src/simcore\_service\_storage/handlers\_simcore\_s3.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fhandlers_simcore_s3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvaGFuZGxlcnNfc2ltY29yZV9zMy5weQ==) | `100.0% <100.0%> (ø)` | | | [...ices/storage/src/simcore\_service\_storage/models.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2UvbW9kZWxzLnB5) | `96.0% <100.0%> (ø)` | | | [...rage/src/simcore\_service\_storage/simcore\_s3\_dsm.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796?src=pr&el=tree&filepath=services%2Fstorage%2Fsrc%2Fsimcore_service_storage%2Fsimcore_s3_dsm.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvc3RvcmFnZS9zcmMvc2ltY29yZV9zZXJ2aWNlX3N0b3JhZ2Uvc2ltY29yZV9zM19kc20ucHk=) | `94.6% <ø> (ø)` | | ... and [665 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5796/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)