Week 35 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• [ ] create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• [ ] run systematically the upgrade on bastion, stats, services, storage, demo nodes
• [ ] check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• [ ] Ensure all borg repositories are being updated

k8s cluster

• [ ] Check Pod errors

  k get pods -A -o wide|grep Error

• [ ] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [ ] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [ ] Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• [ ] Ensure download.kiwix.org stats are being recorded
• [ ] Check whether matomo should be upgraded

Grafana

• [ ] Alert list is normal
• [ ] Zimfarm dashboard is normal
• [ ] Jobs durations dashboard is normal
• [ ] There is no abnormal behaviors on cluster resources consumption
• [ ] All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• [ ] UptimeRobot has no alert
• [ ] zimit backlog is reasonable
• [ ] Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• [ ] Analyze zimit failed tasks
• [ ] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

missed