Week 12 routine - Githubissues

kiwixbot commented 3 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

benoit74 commented 3 months ago

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.9G	27G	26%	-
stats	233G	97G	120G	44%	-5G
services	456G	226G	206G	53%	+1G, +1%
storage	33T	19T	13T	61%	+1%

⚠️ Significant burst of activity last week recorded in stats.kiwix.org for https://download.openzim.org/ ?

zimit

2x Python 403 (uniontestprep.com: Cloudflare, biggerthansnowden.com: Cloudflare, healthtohome.ru, reddit.com) 1x Python name resolution failed (www.anatomy.con, www.portalursinhoscarinhosos.com.br) 1x Python 400 (https://ofs-fe1851d3a5455b02e58aa7e6ea4cc30c.read.overdrive.com/, real issue) 1x Python read timeout (https://kingofmanga.com/, blocked on my desktop as well) 1x Python ssl verification failed (https://www.recipesource.com/special-diets/vegetarian/indexall.html) 1x Python connection timeout (https://risidata.com, same issue on my machine + https://www.callofduty.com) 1x Python connection reset by peer (https://www.ungafakta.se/, same issue on my machine)

benoit74 commented 3 months ago

Nota: I altered youzim.it configuration to go:

from 3 CPU, 6 GB memory per task
to 2 CPU, 5 GB memory per task

Since worker has 6 CPU, 15GB per task, we will start 3 tasks in parallel instead of 2.

Ongoing/queued tasks have been updated to make this change quick.

Youzim.it queue size is currently at 159 items, older task is 24 days old.

kiwix / operations