scality / cloudserver

Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.
https://www.zenko.io/cloudserver
Apache License 2.0
1.71k stars 241 forks source link

Bugfix/cldsrv 553 #5569

Closed williamlardier closed 5 months ago

williamlardier commented 5 months ago

Fixes a limitation during a corner case, when restoring an object within account/bucket with enabled exceeded quota:

This PR introduces a way to detect DMF calls (the specific header) and, in this case, only check the quotas, without increasing the in-flights.

We compare the quota with the current bytes usage minus the reserved space. This ensure that any already-exceeded quota won't block the operation, as the space was reserved.

This solution is not optimal as it is expected that the quota will never exceed the limit by a lot: at the end, we might allow all objects to be restored. But this case can happen only if some in-flights were lost due to the limitations of the metrics update frequency.

A better solution to account for our limitation would use the total number of reserved space bytes, and perform the following operation:

utilizationMetrics - restoringBytes + currentObjectBytes <= quota

In this case, we can better account for the real reserved space: we keep track of the reserved bytes, and thus restores will not exceed space already used by other operations before.

This will be done in a second iteration: for now, we consider that the reserved space is reserved, and lost in-flights are part of the known limitations of the feature.


Alternatively, we can decide to not subtract the current bytes with the current object to restore' size: but in this case, any quota exceeded will deny the request.

bert-e commented 5 months ago

Hello williamlardier,

My role is to assist you with the merge of this pull request. Please type @bert-e help to get information on this process, or consult the user documentation.

Available options name | description | privileged | authored ------ | ------------ | ------------ |---------- `/after_pull_request` | Wait for the given pull request id to be merged before continuing with the current one. | | `/bypass_author_approval` | Bypass the pull request author's approval | :star: | `/bypass_build_status` | Bypass the build and test status | :star: | `/bypass_commit_size` | Bypass the check on the size of the changeset ```TBA``` | :star: | `/bypass_incompatible_branch` | Bypass the check on the source branch prefix | :star: | `/bypass_jira_check` | Bypass the Jira issue check | :star: | `/bypass_peer_approval` | Bypass the pull request peers' approval | :star: | `/bypass_leader_approval` | Bypass the pull request leaders' approval | :star: | `/approve` | Instruct Bert-E that the author has approved the pull request. | | :writing_hand: `/create_pull_requests` | Allow the creation of integration pull requests. | | `/create_integration_branches` | Allow the creation of integration branches. | | `/no_octopus` | Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead | | `/unanimity` | Change review acceptance criteria from `one reviewer at least` to `all reviewers` | | `/wait` | Instruct Bert-E not to run until further notice. | |
Available commands name | description | privileged ------ | ------------ | ------------ `/help` | Print Bert-E's manual in the pull request. | `/status` | Print Bert-E's current status in the pull request ```TBA``` | `/clear` | Remove all comments from Bert-E from the history ```TBA``` | `/retry` | Re-start a fresh build ```TBA``` | `/build` | Re-start a fresh build ```TBA``` | `/force_reset` | Delete integration branches & pull requests, and restart merge process from the beginning. | `/reset` | Try to remove integration branches unless there are commits on them which do not appear on the source branch. |

Status report is not available.

bert-e commented 5 months ago

Jira issue not found

The Jira issue CLDSRV-553 was not found.

williamlardier commented 5 months ago

See https://github.com/scality/cloudserver/pull/5570