Week 38 routine - Githubissues

kiwix / operations

Kiwix Kubernetes Cluster

http://charts.k8s.kiwix.org/

5 stars 0 forks source link

Week 38 routine #122

Closed kiwixbot closed 9 months ago

kiwixbot commented 9 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

benoit74 commented 9 months ago

[NODE kbastion]
/dev/sda1        37G  5.1G   30G  15% /
/dev/sda15      124M   12M  113M  10% /boot/efi
---
[NODE kstats]
/dev/md1        233G   90G  132G  41% /
/dev/md0        485M   80M  376M  18% /boot
---
[NODE kservices]
/dev/md1        456G  181G  252G  42% /
/dev/md0        485M   80M  376M  18% /boot
---
[NODE kstorage]
/dev/md1         33T   19T   13T  59% /
/dev/md0        485M   80M  376M  18% /boot

Upgrade : libwebp6 ((0.6.1-2.1+deb11u2) over (0.6.1-2.1+deb11u1)) on all except bastion

Youzimit backlog is big, but decreasing, see https://github.com/openzim/zimfarm/issues/819 for details of the issue.