Week 16 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [ ] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	11G	25G	30%	-
stats	/	233G	102G	119G	47%	-
services	/	456G	243G	190G	57%	+1G, +1%
storage	/	33T	21T	11T	66%	+1%
demo	/	916G	88G	782G	11%	don't care
imager-worker	/	1.9T	873G	916G	49%	don't care
sisyphus	/	233G	11G	211G	5%	-
ondemand	/	25G	9.5G	15G	41%	-
ondemand	/data	216G	50G	155G	25%	-

Package updates

• demo:
  • containerd.io (1.6.31-1) over (1.6.28-2)
  • docker-ce-cli (5:26.0.1-1~debian.10~buster) over (5:26.0.0-1~debian.10~buster)
  • docker-ce (5:26.0.1-1~debian.10~buster) over (5:26.0.0-1~debian.10~buster)
  • docker-ce-rootless-extras (5:26.0.1-1~debian.10~buster) over (5:26.0.0-1~debian.10~buster)
  • docker-compose-plugin (2.26.1-1~debian.10~buster) over (2.25.0-1~debian.10~buster)
• imager-worker
  • containerd.io (1.6.31-1) over (1.6.28-2)
  • docker-ce-cli (5:26.0.1-1~debian.12~bookworm) over (5:26.0.0-1~debian.12~bookworm)
  • docker-ce (5:26.0.1-1~debian.12~bookworm) over (5:26.0.0-1~debian.12~bookworm)
  • docker-ce-rootless-extras (5:26.0.1-1~debian.12~bookworm) over (5:26.0.0-1~debian.12~bookworm)
  • docker-compose-plugin (2.26.1-1~debian.12~bookworm) over (2.25.0-1~debian.12~bookworm)
• sisyphus + ondemand
  • containerd.io (1.6.31-1) over (1.6.28-2)
  • docker-ce-cli (5:26.0.1-1~debian.11~bullseye) over (5:26.0.0-1~debian.11~bullseye)
  • docker-ce (5:26.0.1-1~debian.11~bullseye) over (5:26.0.0-1~debian.11~bullseye)
  • docker-ce-rootless-extras (5:26.0.1-1~debian.11~bullseye) over (5:26.0.0-1~debian.11~bullseye)
  • docker-compose-plugin (2.26.1-1~debian.11~bullseye) over (2.25.0-1~debian.11~bullseye)

Youzim.it failed task

• 1x URL typo: https://wwww.givex.com
• 1x Python 500 (real error): https://berg.ooo/
• 13x Python 403 (all OK on my machine)
  • 4x https://metanit.com/
  • 5x https://www.reddit.com/
  • 1x https://www.deviantart.com/
  • 1x https://www.cartoonnetwork.com
  • 1x https://unlockt.me/v/0f4be7054e
  • 1x https://minecraft.wiki
• 4x Python SSL issue (all OK on my machine)
  • 2x https://wiki.project1999.com/
  • 2x https://legalacts.ru/
• 2x Python 404 (all OK on my machine)
  • 1x https://marathonphotos.live/Event/Sports%2FCPUK%2F2024%2FLondon%20Landmarks%20Half%20Marathon/9538/
  • 1x https://dragoborne.fandom.com/wiki
• 1x Python connection timeout (all OK on my machine)
  • 1x https://www.tirerack.com/content/tirerack/desktop/en/all-brands.html
  • 1x https://www.carolwright.com/
  • 1x https://www.carrefourqatar.com
• 21x wp1 errors
  • 20x HTTP 400 errors on few articles (retried many times)
  • 1x failed to retrieve article list from S3
• 2x Navigation timeout of 90s
  • 1x https://wamboala.substack.com/archive, ok on my machine
  • 1x http://olegdal.ru/, ok on my machine
• 1x OSError: Unable to write ZIM file at /output/Turok Sanctum 04/13/2024_115fb1f0.zim
  • 🎁 I opened https://github.com/openzim/warc2zim/issues/232
• 1x Crawl seed page load error: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjbh-2578SFAxUuQjABHfJtDoY4ChAWegQIBBAB&url=https%3A%2F%2Fwww.reddit.com%2Fr%2Froosterteeth%2Fcomments%2F2f6rnc%2Fso_all_these_pictures_of_celebs_leak_and_yet%2F&usg=AOvVaw2OA24D7TXIW1tw_0s-GGvq&opi=89978449