frappe / press

Full service cloud hosting for the Frappe stack - powers Frappe Cloud
https://frappe.cloud
GNU Affero General Public License v3.0
259 stars 187 forks source link

Avoid rate limits for OCI block volume snapshots #2076

Closed adityahase closed 1 month ago

adityahase commented 1 month ago

With ~35 machines (1 block volume attached to each). We are hitting rate limits for volume snapshots on OCI.

This breaks somewhere around 30 snapshots. Ref https://frappecloud.com/app/dashboard-view/Snapshots

{
    "target_service": "blockstorage",
    "status": 429,
    "code": "TooManyRequests",
    "opc-request-id": "XXXXXXX",
    "message": "Too many requests for the user",
    "operation_name": "create_boot_volume_backup",
    "timestamp": "2024-08-18T18:47:54.265715+00:00",
    "client_version": "Oracle-PythonSDK/2.116.0",
    "request_endpoint": "POST https://iaas.me-jeddah-1.oraclecloud.com/DDDDDDDD/bootVolumeBackups",
    "logging_tips": "To get more info on the failing request, refer to https://docs.oracle.com/en-us/iaas/tools/python/latest/logging.html for ways to log the request/response details.",
    "troubleshooting_tips": "See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_429__429_toomanyrequests for more information about resolving this error. Also see https://docs.oracle.com/iaas/api/#/en/iaas/20160918/BootVolumeBackup/CreateBootVolumeBackup for details on this operation's requirements. If you are unable to resolve this blockstorage issue, please contact Oracle support and provide them this full error message.",
}

References: