✨ DevOps: Reduce and finetune resources usage

[edulix]

Preflight checklist

• [X] I could not find a solution in the existing issues, docs, nor discussions.

Suggestion

Right now our 2nd-gen environment use too much resources. We should reduce and finetune resources usage.

Possible Implementation

No response

Tasks

# Tasks
- [x] Identify approximate current costs per environment and put it in a spreadsheet
- [x] Talk with Angel about this
- [x] Identify manners to reduce costs, list them
- [ ] Tag Environments to identify resource usage by environment
- [ ] Apply cost reduction tips identified on previous issue

# Main PRs
- [ ] https://github.com/sequentech/gitops/pull/56
- [ ] https://github.com/sequentech/gitops/pull/57
- [ ] https://github.com/sequentech/gitops/pull/58
- [ ] https://github.com/sequentech/gitops/pull/59
- [ ] https://github.com/sequentech/gitops/pull/60

Additional context

No response

[torgus]

Approximate cost per enironment.

https://docs.google.com/spreadsheets/d/1_kwXLa6rwZDf-QD3u16W5Wot3r8ZGb-nlIBZMb5YXQs/edit?usp=sharing

[torgus]

Recomendations to reduce cost:

• Tag all resources, to identify exact cost per environment, and no approximate costs.
• Move Rabbit MQ to kubernetes to reduce costs, we can achieve same redundacy using multiples Azs and nodes.
• Reduce size of postgresql, cpu is around 10% all the time, so a t3 instance probably can do the job keeping cpu below 20% ( baseline for t type instances)
• Clean Unused Resources on us-east-2 region , eu-west-2 region ( i.e unused nat gateways and balancers)
• Try to limit pod resources, to avoid autoscaling as much as possible
• Reserve instances.

[angelAtSequent]

From Kubernetes i would say:

• Adopt karpenter
• Adopt descheduler
• Adopt kube-downscaller
• Avoid using tX instances (t2, t3...) (cpu throthelling)
• Adopt HPA/VPA where required
• Know kubernetes storage restrictions (while running stateful workloads like rabbitmq)
• Single node kubernetes for development/demo purposes. (ec2 with kind or k3s)