Automate Quota Approval Process spike

[w8896699]

Describe the issue Objective: Investigate and implement automation of the quota approval process in Kubernetes to streamline operations, ensure consistent policy application, and reduce manual overhead.

Additional context Add any other context, attachments or screenshots

Steps to Complete

• [x] Create Sysdig Template

  • Develop a template in Sysdig to capture necessary data for automating quota approvals.

• [ ] Share Template with Sysdig Teams

  • Distribute the newly created Sysdig template to all relevant Sysdig teams for implementation and feedback.

• [x] Document Quota Request Process

  • Create comprehensive documentation detailing the process of requesting quotas, including prerequisites and steps to follow.

• [x] Add Resource Quota Management Guide

  • Compile a detailed guide on managing resource quotas in Kubernetes, covering best practices, common issues, and troubleshooting.

• [x] Investigate Red Hat Cost Management Optimization

  • Explore features and tools provided by Red Hat for cost management to determine applicability to quota management.

• [x] Explore Sysdig Cost Advisor

  • Evaluate Sysdig Cost Advisor for potential benefits in cost management and integration into the quota approval process.

• [x] Share Findings with Registry Team

  • Present findings from the Red Hat cost management investigation to the registry team and create follow-up tasks.

• [x] Discuss with Platform Admin Team

  • Share insights from both the Red Hat and Sysdig cost management tools with the platform administration team for decision-making.

• [ ] Team Discussion on Findings

  • Organize a meeting to discuss all findings with the team and decide on the next steps forward.

Definition of done

• Improved efficiency in quota management.
• Better integration of cost management tools.
• Enhanced documentation for internal processes.

[w8896699]

Redhat API: https://console.redhat.com/docs/api/cost-management/v1#operations-Optimizations-getRecommendationList

[w8896699]

sysdig dashboard template: https://app.sysdigcloud.com/#/dashboards/410674?last=3600&scope=kubernetes.cluster.name%20as%20%22cluster%22%20in%20%3F%28%22silver%22%29%20and%20kubernetes.namespace.name%20as%20%22namespace%22%20in%20%3F

[w8896699]

Managing resource quotas in Kubernetes: A complete guide¶ https://developer.gov.bc.ca/docs/default/component/platform-developer-docs/docs/app-monitoring/managing-resource-quotas-in-kubernetes/

[w8896699]

quota approve process: https://developer.gov.bc.ca/docs/default/component/platform-developer-docs/docs/app-monitoring/resource-monitoring-dashboards/#prepare-request-for-quota-increase

[w8896699]

Linked to this ticket: https://app.zenhub.com/workspaces/app-dev-team-646fdb4400ed8c0021aceeb4/issues/gh/bcgov/platform-services-registry/3030

[w8896699]

Option 1,

Billy contribute to registry for automated quota approve process

Option 2

Billy create a posudocode and let registry app team to develop the feature

Option 3

Billy create ts script to registry team

Option 4

Billy will create a separate server to run the task

• Ask if there’s option to embed red-hat resources utilization into console instead of registry, because registry is the last step they are doing
• Ask if those red-hat resources utilization logic, documentation about how those recommendation

Next step

1. auto create sys dig-team in tools, ignore watch for argoCD
2. copy the dashboard quota