Week 09 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	8.7G	27G	25%	-
stats	233G	102G	120G	47%	-1G
services	456G	220G	213G	47%	+3G
storage	33T	17T	15T	55%	-

zimit

• Very long waiting list (about 100). Has there been any publicity in the last couple days?
• Opened https://github.com/openzim/zimfarm/issues/924
• 1 x Crawler Seed Page Load Error
• 4 x Python 403 Client Error: Forbidden
• 1 x Python 403 Client Error: Blocked
• 17 x https://github.com/openzim/zimfarm/issues/925
• 1 x Python argument --waitUntil: invalid choice: ''

---

• ⚠️ backlog is long but there is no blocking task.
• https://github.com/openzim/zimfarm/issues/925 very frequently (not systematically though) fails otherwise successful runs.

[rgaudin]

Awaiting kube-state-metrics upgrade before upgrading the cluster

[benoit74]

The fact that https://github.com/openzim/zimfarm/issues/925 very frequently (not systematically though) fails otherwise successful runs is probably linked to requested tasks that have been created before the change. I will nevertheless patch the code to fix this.

[rgaudin]

Ah right ; I didn't think about it. Most likely