Week 13 routine - Githubissues

kiwixbot commented 3 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 3 months ago

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.9G	27G	26%	-
stats	233G	97G	124G	44%	-4G
services	456G	230G	203G	54%	+3G
storage	33T	19T	13T	61%	-

@benoit74 can you maybe update https://github.com/kiwix/k8s/wiki/Monitoring so that we easily know where/how to check the compatibility so that we eventually upgrade the kube. It's been a while now

zimit

Pipe is now empty
(1) 401 Client Error: Unauthorized
(10) 403 Client Error Forbidden
(1) 403 Client Error: Blocked
(1) 429 Client Error: Too Many Requests (on seed)
(1) Exceeded 30 redirects
(1) Connection refused
(1) Navigation timeout of 90000 ms exceeded
(1) failed to connect
(1) Connection timed out
(2) Read timed out
(1) argument --waitUntil: invalid choice: '30000'
(2) got canceled after two hours by periodic_tasks. not sure why crawler didn't stop before
(1) CERTIFICATE_VERIFY_FAILED

benoit74 commented 2 months ago