Week 49 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal (look after last week data)
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [ ] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Filesystem	Size	Used	Avail	Use%	Use change
bastion	37G	7.8G	28G	23%	+0.1G ; +1%
stats	233G	93G	128G	43%	+1G ; +1%
services	456G	202G	231G	47%	-
storage	33T	19T	13T	61%	-

• Upgrade libnghttp2-14:amd64 (1.43.0-1+deb11u1) over (1.43.0-1) on stats, services and storage nodes
• cert-manager and cert-manager-cainjector have restarted 4 days ago
• both library-data-backend-deployment have restarted in the last 7 days
• Matomo updated to 4.16.0

Zimit

• 1x Python Read timed out
• 5x Python HTTP 403 Forbidden
• Many times (same user/issue) on WP1 : Title: must NOT have more than 30 characters
• 2x Python NameResolutionError
• Many times (same user/issue) on WP1 : HTTP 400 Invalid Title name
• 1x Browsertrix Invalid Seed (while page is OK)
• 1x Python NewConnectionError
• 1x Browsertrix timeout (someone trying to zim http://download.kiwix.org/zim/wikipedia_ar_all.zim ... 🤓)