search

edgexr / edge-cloud-platform

Apache License 2.0
1 stars 0 forks source link

infra (openstack): monitor network quotas #164

Open gainsley opened 1 year ago

gainsley commented 1 year ago

Openstack implements quotas on resources we can use. We have some monitoring and alerting around cpu/mem quotas, but none on network quotas. We recently hit errors on deploying AppInsts because we hit a security group rule quota limit:

    "message": "Encountered failures: Create App Inst failed: Heat Stack failed: Resource CREATE failed: OverQuotaClient: resources.win1010-mytestorg.*****.eu.app.****.edgexr.org-sg: Quota exceeded for resources: ['security_group_rule'].\nNeutron server returns request_ids: ['req-77ff1c7a-e6de-4d7e-b7b6-c436c1cd86e9']"

Unfortunately at this time it appears we can't even see what those quota limits are.

# openstack quota list --network
You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-e4cf610b-d760-42d9-b301-3aadba75687c)

We should add listing quotas to the list of openstack commands needed for onboarding a new openstack cloudlet.

We should add monitoring and alerting when we are close to using up any of our network quotas.

gainsley commented 1 year ago

Update: we do have permissions to see quotas, correct command is openstack quota show --fit