Week 34 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

@rgaudin I will take this one also since you did it twice during my holidays

[rgaudin]

👍

[benoit74]

Free space:

Node        Filesystem      Size  Used Avail Use% 
kstorage   /dev/md1         33T   18T   13T  59% 
kservices  /dev/md1        456G  168G  265G  39% 
kstats     /dev/md1        233G   89G  133G  40% 
bastion    /dev/vda1        38G   21G   16G  58% 

[benoit74]

Nodes (except bastion which was already up-to-date):

• upgrade linux-libc-dev:amd64 (5.10.191-1) over (5.10.179-5)

[benoit74]

We continue to routinely have issues with backup jobs that fails at retrieving unlocking bitwarden vault, and then succeed few seconds later

[benoit74]

Issue with cert-manager continues as well.

[benoit74]

Had to cancel https://farm.youzim.it/pipeline/7023934f-acfa-4d59-9887-933eddd2095d which was running for 7 days and reducing youzimit processing capability

[rgaudin]

Had to cancel https://farm.youzim.it/pipeline/7023934f-acfa-4d59-9887-933eddd2095d which was running for 7 days and reducing youzimit processing capability

This is unexpected because:

• we're now up to date with upstream crawler so crawler bugs must be assessed and upstream ticker open if required
• this happened in the zimit (python) phase (visible from the log format) so we ought to fix it.
• There's a recurring error log about progress that gives us a clue about what might have gone wrong

[rgaudin]

So you should open a ticket on zimit

[benoit74]

Done: https://github.com/openzim/zimit/issues/205