Week 46 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• [x] create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• [x] run systematically the upgrade on bastion, stats, services, storage, demo, mirrors-qa nodes
• [x] check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• [x] Ensure all borg repositories are being updated

k8s cluster

• [x] Check Pod errors or in CrashLoopBackoff

  k get pods -A -o wide|grep -E 'Error|Crash'

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [x] Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Jobs durations dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption
• [x] All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• [x] Analyze zimit failed tasks and document bad domains in our WIP blacklist
• [x] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	16G	20G	44%	-
stats	/	233G	119G	103G	54%	+2G
services	/	456G	334G	99G	78%	+1G
storage	/	147G	65G	75G	43%	+5G
storage	/data	30T	14T	15T	49%	-
imager-worker	/	1.9T	452G	1.4T	26%	don't care
sisyphus	/	233G	15G	206G	-	don't care
ondemand	/	25G	9.7G	14G	42%	-
ondemand	/data	216G	199M	205G	1%	don't care
mirrors-qa	/	38G	4.4G	32G	13%	-
demo	/	40G	9G	29G	25%	-2G
demo	/data	1.8T	968G	695G	59%	don't care

Misc

• rsyncd service in zim NS restarted at 2024/11/08 19:02:02 without any message in the logs ...
• Cloud Signing: Unused Signings: 899
• ZIMit has 28 entries in the pipe, oldest task is 16 hours old

zimit

• (1) x net::ERR_CONNECTION_REFUSED (Seed Page Load Failed)
• (1) x net::ERR_ADDRESS_UNREACHABLE (Seed Page Load Failed)
• (2) x net::ERR_TIMED_OUT (Seed Page Load Failed)
• (4) x net::ERR_HTTP2_PROTOCOL_ERROR (Seed Page Load Failed)
• (1) x net::ERR_NAME_NOT_RESOLVED (Seed Page Load Failed)
• (3) x Navigation timeout of 90000 ms exceeded (Seed Page Load Failed)
• (3) x https://github.com/openzim/warc2zim/issues/383
• () x https://github.com/openzim/warc2zim/issues/380
• (1) x https://github.com/openzim/warc2zim/issues/331 iso88591
• (1) x https://github.com/openzim/warc2zim/issues/420 (NEW, maybe different than #331)
• (1) x https://github.com/openzim/warc2zim/issues/421 (NEW)
• (6) x https://github.com/webrecorder/browsertrix-crawler/issues/719 (NEW)

[rgaudin]

rsyncd service in zim NS restarted at 2024/11/08 19:02:02 without any message in the logs ...

Exit Code: 137