Week 10 routine - Githubissues

kiwixbot commented 4 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[ ] UptimeRobot has no alert
[ ] youzim.it backlog is reasonable
[ ] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

benoit74 commented 4 months ago

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.7G	27G	26%	+1%
stats	233G	103G	119G	47%	+1G
services	456G	224G	209G	52%	+4G
storage	33T	18T	13T	59%	⚠️ +1T ; +4%

⚠️ storage filesystem usage => linked to all TED videos by topics (1.2T so far, not yet complete)

Bastion update: scaleway-ecosystem (0.0.6-13~debian11) over (0.0.6-11~debian11)

youzim.it

⚠️ backlog is even bigger than last week (118 tasks ATM) => should we start another worker / rework task assignment in zimit-frontend / replace sisyphus by a bigger machine?

Failures:

1x Python SSLCertVerificationError (www.bacalhoa.pt, www.houseofpaa.com)
1x Python SSL Handshake failure (1100ad.com)
tens x Python 403 (1x maps.plebmasters.de, 2x www.reddit.com, tens x gearspace.com)
4x argument --waitUntil: invalid choice
1x Python 500 (wikiwel.com)
1x Python 404 (real ...)
1x Python 400 (real ...)
1x Python NameResolutionError (real ...)
1x Python Read timeout (real ...)
1x Main page not found in warc (https://github.com/openzim/warc2zim/issues/203)

⚠️ I confirm that last week issue about logs formatting on youzim.it is something new

rgaudin commented 3 months ago

@benoit74 you never closed this one…

benoit74 commented 3 months ago

Let's close it now then ... I don't recall exactly but it looks like I did everything ... sorry anyway

kiwix / operations