Week 06 routine - Githubissues

kiwixbot commented 4 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 4 months ago

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.5G	27G	25%	-0.1G
stats	233G	93G	129G	42%	-1G
services	456G	193G	240G	52%	-31G
storage	33T	17T	15T	54%	-3TB

Increase in free space on services node is due to purge actions taken in #163 For storage node I believe @benoit74 cleaned-up the folder(s) where we stored ZIM that we delete.

benoit74 commented 4 months ago

For storage node I believe @benoit74 cleaned-up the folder(s) where we stored ZIM that we delete. Yes, I did "a bit" of cleanup

Matomo updated to 5.0.2 (from 5.0.1)

zimit

Same error on same domain is counted only once.

1x Invalid whitespace in header => https://github.com/openzim/zimit/issues/281
2x Python CERTIFICATE_VERIFY_FAILED (www.vasp.at, www.moneyinstructor.com)
1x Python SSLV3_ALERT_HANDSHAKE_FAILURE (panzer-war.com)
2x Python Connection refused (www.facebook.com, web.archive.org)
2x Python 403 error (news.ycombinator.com, www.reddit.com)
1x Python timeout (www.lansforsakringar.se, wikilivres.ru)
1x 90000ms timeout on Seed page (www.lansforsakringar.se)
2x Python failed to resolve (safudex.cz, ww.ired.team)
1x Python 503 (silverorchidphotography.com)
1x Python 404 (translated.turbopages.org)
1x Python read timeout (www.tesla.com)

kiwix / operations