Week 21 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	12G	24G	33%	-
stats	/	233G	103G	119G	47%	+2G
services	/	456G	273G	160G	64%	+1G
storage	/	33T	21T	11T	67%	-
demo	/	916G	32G	838G	4%	don't care
imager-worker	/	1.9T	885G	905G	50%	don't care
sisyphus	/	233G	15G	206G	7%	-
ondemand	/	25G	9.7G	14G	42%	+0.1G
ondemand	/data	216G	63G	142G	31%	+1G

Misc

• docker update for several machines: hotspot demo, imager worker, youzim.it worker, and ondemand worker
• zimfarm watcher restarted once. Issue is not clear but logs of that run indicate networking issues (failed to resolve archive.org)
• many cms issues ; restarted

zimit

• queue is shrinking: only 384 tasks at the moment
• 3 7d old dead tasks. maybe from last routine. canceled them
• (13) 403 Client Error: Forbidden
• (2) Navigation timeout of 90000 ms exceeded
• (2) [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
• (1) Connection refused
• (2) 404 Client Error: Not Found
• (1) 403 Client Error: Blocked
• Opened https://github.com/openzim/zimit/issues/299
• (1) 405 Client Error: Not Allowed
• (1) 451 Client Error: Unavailable For Legal Reasons
• (1) Read timed out
• (1) connect timeout
• (1) [Errno 111] Connection refused