Week 27 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Jobs durations dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	12G	24G	33%	-
stats	/	233G	104G	118G	47%	+1G
services	/	456G	283G	150G	66%	+4G
storage	/	33T	22T	9.9T	69%	+1T, +1%
demo	/	916G	126G	744G	15%	don't care
imager-worker	/	1.9T	999G	790G	56%	don't care
sisyphus	/	233G	19G	203G	8%	-
ondemand	/	25G	10G	14G	43%	-
ondemand	/data	216G	3.7G	201G	2%	don't care

zimit

• pipe has only 53 tasks
• 4 tasks failed due to docker update on the worker: https://farm.zimit.kiwix.org/pipeline/b60b5c44-5bf3-4435-8a21-94434c425e97, https://farm.zimit.kiwix.org/pipeline/7d05cf54-2f45-438d-ab8c-24aa711b77ca, https://farm.zimit.kiwix.org/pipeline/258b3d47-d6a4-44ae-a2ff-77a37561f14b, https://farm.zimit.kiwix.org/pipeline/4501f197-c5b2-4a82-86aa-50b9eb5c531e
• (1) NameResolutionError
• (4) Page Load Timeout
• (4) INTERNAL ERROR (puppeteer)
• (1) Navigation timeout of 90000 ms exceeded
• (3) ValueError: Title is too long.
• (16) https://github.com/openzim/zimit/issues/203
• (3) https://github.com/openzim/warc2zim/issues/331
• (1) https://github.com/openzim/zimit/issues/329
• (1) https://github.com/openzim/warc2zim/issues/332

[benoit74]

Many Zimfarm tasks have failed to due huge pipe of youtube tasks + pixelmemory has been blacklisted from yt-dlp, causing many tasks to fail in a row. Luckily, worker owner (Kevin) has quickly stopped his worker from processing youtube tasks. Ravan informed on Thursday evening, she said she will restart what needs to be. Still not done for many schedules.

[benoit74]

Library generation job durations are still highly variable, this is now a known issue but it did not disappeared by magic ...