Week 22 routine - Githubissues

kiwix / operations

Kiwix Kubernetes Cluster

http://charts.k8s.kiwix.org/

5 stars 0 forks source link

Week 22 routine #198

Closed kiwixbot closed 4 weeks ago

kiwixbot commented 1 month ago

Infra

[ ] Check nodes free space
```
df -h / && df -h /data
```
[ ] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[ ] Ensure all borg repositories are being updated
[ ] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[ ] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[ ] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[ ] Upgrade k8s if applicable

Stats

[ ] Ensure download.kiwix.org stats are being recorded
[ ] Check whether matomo should be upgraded

Grafana

[ ] Alert list is normal
[ ] Zimfarm dashboard is normal
[ ] Mirrorbrain dashboard is normal
[ ] There is no abnormal behaviors on cluster resources consumption

Projects

[ ] UptimeRobot has no alert
[ ] zimit backlog is reasonable
[ ] Analyze zimit failed tasks
[ ] PRs awaiting your review

Security

[ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

benoit74 commented 4 weeks ago

Dropped out of my plate for some mysterious unjustifiable reasons, will do #200 right now.