Week 05 routine - Githubissues

kiwixbot commented 5 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 5 months ago

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.4G	27G	24%	+0.1G
stats	233G	94G	128G	43%	-2G
services	456G	224G	209G	52%	+5G
storage	33T	20T	12T	62%	-

Most of the increase in services is due to the layers. @benoit74 let's discuss this today.

zimit

(6) Python 403 Client Error: Forbidden
(1) Python 403 Client Error: Blocked
(1) Python 404 Client Error: Not Found
(1) Python 503 Server Error: Service Unavailable
(1) Python 525 Server Error (525 SSL handshake failed)
(1) Python ConnectTimeoutError
(2) Crawler Page Load Timeout
(1) Crawler Seed Page Load Error from 403
(1) recipe issue (--waitUntil value)

benoit74 commented 5 months ago

We have configured (image) garbage collection on services node to clean-up unused images.

In /var/lib/kubelet/kubelet.conf, we have configured (no values where present):

imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 50

Disk usage went down to 44%

Documentation used: