Week 50 routine - Githubissues

kiwixbot commented 6 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal (look after last week data)
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 6 months ago

Filesystem      Size  Used Avail Use% Mounted on
bastion        37G  7.8G   28G  23% /
stats        233G   95G  127G  43% /
services        456G  202G  231G  47% /
storage         33T   19T   13T  61% /

storage node took forever to display disk stats
mirrorbrain dashboard shows an abnormally long duration
zimfarm dashboard indicates 173 502 errors

zimit

Opened https://github.com/openzim/warc2zim/issues/132
(1) 404 to an actual 404
(3) 403 to real content
(1) ConnectionResetError(104, 'Connection reset by peer')
(5) Page Load Timeout
(2) failed to connect on a missing websites
(1) Invalid leading whitespace: URL is not HTML but an XML file

benoit74 commented 6 months ago

Regarding mirrorbrain update db job which takes too long, I created a small tool to analyze mb update db job logs: https://gist.github.com/benoit74/fb95eff59fff6fa094282b1c63ddefa3 (of a single job)

The culprit is mirror-sites-in.mblibrary.info which now regularly takes hours (3-4 hours) to be scanned (instead of ~ 10-15 minutes usually)

kiwix / operations