Week 32 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• [x] create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• [x] run systematically the upgrade on bastion, stats, services, storage, demo nodes
• [x] check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus, worker-mirrors-qa)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• [x] Ensure all borg repositories are being updated

k8s cluster

• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [x] Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Jobs durations dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	14G	22G	40%	+2G
stats	/	233G	111G	111G	51%	+2G
services	/	456G	299G	134G	70%	+5G
storage	/	33T	22T	9.1T	71%	+0.2T
demo	/	916G	112G	759G	13%	don't care
imager-worker	/	1.9T	449G	1.4T	26%	don't care
sisyphus	/	233G	22G	199G	10%	don't care
ondemand	/	25G	11G	14G	43%	don't care
ondemand	/data	216G	41G	165G	20%	don't care
mirrors-qa	/	38G	2.8G	33G	8%	-

• Grafana Zimfarm dashboard is not working anymore due to https://github.com/kiwix/operations/issues/229
• jobs (mirrorbrain db update + libraries generation) seems to get a bit slower over the last 3 days ; this confirms that problem on storage node is still present (even if there is no more incident for now)

zimit

• pipe is 12 requested tasks, oldest one is 2 hours ago => OK
• 1 x https://github.com/openzim/warc2zim/issues/369
• 7 x https://github.com/openzim/warc2zim/issues/332
• 6 x https://github.com/openzim/warc2zim/issues/331
• 1 x Page Load Timeout (ERR_HTTP2_PROTOCOL_ERROR)
• 3 x Page Load Timeout (ERR_CONNECTION_REFUSED)
• 1 x Page Load Timeout (ERR_ABORTED)
• 1 x ERR_HTTP_RESPONSE_CODE_FAILURE on seed page
• 1 x Unable to get new page, browser likely crashed
• 3 x ValueError: Title is too long.
• 1 x Execution context was destroyed
• 5 x Page Load Timeout (Navigation timeout of 90000 ms exceeded)