Week 18 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	11G	25G	30%	-
stats	/	233G	101G	121G	46%	+1G
services	/	456G	259G	174G	60%	+9G
storage	/	33T	21T	11T	67%	+1%
demo	/	916G	32G	838G	4%	don't care
imager-worker	/	1.9T	843G	947G	48%	don't care
sisyphus	/	233G	11G	211G	5%	-
ondemand	/	25G	9.5G	15G	41%	-
ondemand	/data	216G	50G	155G	25%	-

bastion:

• updated scaleway-ecosystem (0.0.6-15~debian11) over (0.0.6-14~debian11)

demo:

• distro-info-data (0.41+deb10u9) over (0.41+deb10u8)
• docker-ce-cli (5:26.1.1-1~debian.10~buster) over (5:26.1.0-1~debian.10~buster)
• docker-ce (5:26.1.1-1~debian.10~buster) over (5:26.1.0-1~debian.10~buster)
• docker-ce-rootless-extras (5:26.1.1-1~debian.10~buster) over (5:26.1.0-1~debian.10~buster)
• docker-compose-plugin (2.27.0-1~debian.10~buster) over (2.26.1-1~debian.10~buster)
• libnghttp2-14:amd64 (1.36.0-2+deb10u3) over (1.36.0-2+deb10u2)

imager-worker:

• docker-ce-cli (5:26.1.1-1~debian.12~bookworm) over (5:26.1.0-1~debian.12~bookworm)
• docker-ce (5:26.1.1-1~debian.12~bookworm) over (5:26.1.0-1~debian.12~bookworm)
• docker-ce-rootless-extras (5:26.1.1-1~debian.12~bookworm) over (5:26.1.0-1~debian.12~bookworm)
• docker-compose-plugin (2.27.0-1~debian.12~bookworm) over (2.26.1-1~debian.12~bookworm)

sisyphus:

• libc6:amd64 (2.31-13+deb11u9) over (2.31-13+deb11u8)
• libc-bin (2.31-13+deb11u9) over (2.31-13+deb11u8)
• libc-l10n (2.31-13+deb11u9) over (2.31-13+deb11u8)
• locales (2.31-13+deb11u9) over (2.31-13+deb11u8)
• docker-buildx-plugin (0.14.0-1~debian.11~bullseye) over (0.13.1-1~debian.11~bullseye)
• docker-ce-cli (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-ce (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-ce-rootless-extras (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-compose-plugin (2.27.0-1~debian.11~bullseye) over (2.26.1-1~debian.11~bullseye)

ondemand:

• docker-buildx-plugin (0.14.0-1~debian.11~bullseye) over (0.13.1-1~debian.11~bullseye)
• docker-ce-cli (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-ce (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-ce-rootless-extras (5:26.1.1-1~debian.11~bullseye) over (5:26.0.1-1~debian.11~bullseye)
• docker-compose-plugin (2.27.0-1~debian.11~bullseye) over (2.26.1-1~debian.11~bullseye) ...

Reported https://github.com/openzim/zimfarm/issues/954

zimit

• 4x Python connection timeout (working on my computer unless specified differently)
  • http://britishpublicschools.over-blog.com/
  • https://bestiary.ca
  • https://starnik.by/pravapis
  • https://collapseos.org/ (not working on my computer)
• 5x Python 403 (working on my computer unless specified differently)
  • https://www.scribblehub.com/read/421879-the-sisters-of-dorley/
  • https://gitlab.com/1NEGROUP/simple-vulkan-library (not working on my computer)
  • https://www.infowars.com/
  • https://www.printables.com/model
  • https://ru.minecraft.wiki
• 1x https://github.com/openzim/zimfarm/issues/955
• 1x Python SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
  • https://www.cottontown.org/Pages/home.aspx (not working on my computer at all now)
• 1x Python 404
  • https://wiibrew.org/wiki/WiiM (same issue on my computer)
• 1x Python 403
  • https://www.reddit.com/r/Piracy/wiki (404 on my computer)
• 1x Python SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate
  • https://birrell.org/andrew/minesweeper/ (same issue on my computer)
• 1x Invalid leading whitespace, reserved character(s), or returncharacter(s) in header value (known issue: https://github.com/openzim/zimit/issues/281)
  • https://bevyengine.org
• 2x Python NameResolutionError error
  • https://armypubs.army.mil/epubs/DR_pubs/DR_a/ARN3039-TC_3-21.76-000-WEB-1.pdf (working on my computer)
  • http://oakwooduniversity.edu (not working on my computer)